Andrew Wegner | Ponderings of an Andy

Management Debt: When Leadership Patterns Rot

2026-04-30T13:15:00-05:00

Introduction¶

Engineering leaders talk about technical debt frequently. We track it, we visualize it, we plan around it, we explain it to executives. There's a related kind of debt that gets a fraction of that attention and, in my experience, does at least as much damage to engineering organizations. I've come to think of it as management debt - the engineering-leadership flavor of what recent research calls organizational debt: the accumulation of outdated structures, policies, and processes that quietly hinder how a company operates.

By management debt I mean the gap between how leadership is supposed to operate and how it actually operates. Decision rights that have quietly migrated to whoever speaks up first. Skip-levels that became calendar holds nobody runs. Quarterly planning sessions that turned into backwards-looking status reviews. Performance feedback that gets saved up for the yearly review cycle because nobody had the harder conversation when a problem occurred.

None of that looks like failure in the moment. It looks like a function nobody changed for three years. It still works, technically. Every change to it just costs more than it should.

How management debt accumulates¶

Most of the management debt I've seen didn't enter the organization through a bad decision. It entered through a deferred one. It entered through a "best at the time" decision that was never revisited.

A skip-level got cancelled because a customer escalation came up. The next month it got cancelled again for similar reasons. After that it quietly fell off the calendar, "until things calm down." But, things never calm down. Six months later the engineering manager is making decisions that the director should be involved in, and the director doesn't know those decisions are being made.

The same pattern shows up in design reviews. A team starts skipping the architecture review for "small" changes. Over time, the definition of "small" expands. A year later, the team is shipping huge changes without review and finding integration issues in production that the review process was specifically designed to catch.

Each of these is defensible at the time. They look like sensible prioritization and efficiency, not debt. The problem is the same as with technical debt. The small concessions accumulate into a system that's worse than the one you designed, and at no single point did anyone make the call to make it worse.

The four patterns I see most¶

I've come to recognize four common patterns. Each has its own diagnostic and its own fix.

Decision rights drift¶

The clearest sign is that you can ask three people who's responsible for a particular call (production deploys, architectural decisions, prioritization of a bug over a feature) and get three different answers. The RACI chart says one thing. Practice has drifted somewhere else, usually because the formal owner delegated informally years ago and the delegation never got written down.

The fix is uncomfortable. List the decisions that matter. Write down who actually owns each one today, not who's supposed to. Then write down who should own it. Where those two lists disagree, you've got a decision to make and to communicate.

I've done this exercise on every leadership team. Every time, it surfaces things the leadership team didn't realize had drifted. Fast-scaling teams are particularly prone to this. Decision rights set when the team was eight people rarely survive contact with thirty.

Ritual decay¶

Skip-levels, design reviews, retrospectives, planning sessions. These are leadership rituals that exist for a reason, and they decay along a predictable path. First the rhythm slips ("let's skip this week"). Then the agenda hollows out ("we don't really have anything to discuss, let's just do round-robin status"). Then the meeting becomes a calendar hold that nobody prepares for and nobody gets value from.

The fix is to either kill the ritual or restore it to its original purpose. If you kill it, write down what it was supposed to do and confirm you're getting that signal another way. Keeping the calendar hold but letting it stay hollow is the worst of both worlds. People look at it, decide leadership doesn't take its own rituals seriously, and stop preparing for the ones that still matter.

Feedback latency¶

Healthy organizations give feedback close to whatever prompted it. Gallup's research is blunt about this: employees who get weekly feedback are far more likely to be engaged than those who only hear from their manager at year-end, and only 14% of employees say their performance review actually inspired them to improve. Organizations carrying management debt batch feedback into year-end performance reviews, where it lands as a list of surprises about events from eight months ago that nobody can do anything about.

This one is the hardest to fix because the people involved have to do something they've been avoiding for a year. The structural piece is making sure feedback has a low-friction venue. Consistent 1:1s, written notes, performance check-ins quarterly rather than annually. The behavioral piece is the one that actually matters: the leader has to commit to delivering feedback soon after whatever prompted it, and their own manager has to check that it's happening.

In my experience, this is where management debt does the most damage to retention. An engineer who hears "you should have been doing X differently for the past year" in a performance review either loses trust in their manager or starts interviewing. Often both. I've seen the same avoidance dynamic show up in unlimited-PTO programs - a perk turns into a liability when the management work behind it never happens.

Meeting metabolism¶

The slowest variant to spot is the gradual expansion of recurring meetings until calendar load makes deep work impossible for the leadership team itself. I like to call this "Calendar Tetris", and symptoms include frequent, overlapping meetings, often of a recurring variety. Each meeting was added for a defensible reason. The aggregate is that an engineering manager spends 32 hours a week in meetings and wonders why nothing strategic is getting done. On globally distributed teams the math gets worse, since synchronous slots compress into a few overlapping hours and "let's just add a recurring sync" becomes the default response to any coordination problem.

The fix is blunt. Cancel everything recurring for two weeks. Re-add only the meetings that someone actively missed. The recurring meetings nobody noticed missing weren't serving the purpose they were created for. They were calendar inertia, and they were eating the leadership team's capacity. Shopify did exactly this at scale in early 2023, deleting roughly 12,000 recurring meetings in a single calendar purge with a two-week cooling-off period before anything could be re-added. Their reported outcome was a 33% drop in time spent in meetings and a 25% increase in completed projects.

I've done this, and it's always refreshing when recurring meetings don't come back. Those weren't as needed as once believed.

How to find your own management debt¶

The diagnostic question I run on myself, and on every leadership team I've inherited, is this: What rituals are we running today that we wouldn't design from scratch if we were starting tomorrow? If the answer is "none," you're either not looking hard enough or you've genuinely just done a refactor.

Another version: where in our leadership operating model are we relying on heroics or institutional memory instead of process? The answers are the line items in your management debt portfolio.

A more concrete version I use with my staff: which decisions in the last quarter took longer than they should have, and why? The "why" almost always points to a piece of management debt. A missing decision right, a stale ritual, a communication channel that nobody runs anymore or that needs the help of management to clean up.

Why this matters more than technical debt¶

Technical debt slows code velocity. Management debt slows everything.

These are decisions that aren't being made on time, feedback that isn't landing, and rituals that aren't producing signal. They show up as engineers who feel like they're working harder for less output, leaders who feel like they're spending the whole day in meetings without making progress, and a strategic plan that everyone agrees with and nobody actually executes.

Engineering leaders are usually fluent in technical debt management. We track it, we explain its cost, we plan its repayment. We need the same discipline for the management practices themselves. Make it visible, talk about it explicitly, refactor it deliberately instead of waiting until the organization is genuinely broken.

Conclusion¶

Across the leadership roles I've held, the leaders I've seen succeed long-term aren't the ones with the most polished initial operating model. They're the ones who are willing to look at how they're actually leading, six or twelve months in, and fix what's drifted.

Management debt is real. It accumulates the same way technical debt does. It costs at least as much, and it doesn't show up in any dashboard you already track. The next time you catch yourself thinking "we should really restart that ritual" or "we should really write down who owns that decision," that's not a nice-to-have on a backlog. That's overdue maintenance, and the longer you defer it the more expensive it gets.

Review of Claude Code for Python Developers from Real Python

2026-04-06T10:30:00-05:00

Introduction¶

If you are in the software industry and haven't started working with an AI assistant by this point in 2026, then you should start getting concerned about your role in the company you work at. Throughout the first few months of 2026, there have been several large layoffs by major corporations. Companies like Block and Oracle cited AI as the driver.

AI is here and as of today, it's the least capable it will ever be, because each day it gets better. I've been exploring it since ChatGPT came out and it's impact on interviews. I've written about it multiple times recently. AI's impact on junior developers is particularly important because I'm already seeing this in my role.

My teams have been utilizing a handful of AI assistants over the past year and making amazing improvements to our workflows. From things as simple as reducing the SDLC cycle time for major features to triage of support items, the impact has been dramatic.

I feel like I've only touched the surface - not even scratched it - touched the surface of how to better use AI tooling, and I've been using it a lot. So, I looked for a workshop to increase my knowledge. As a subscriber to Real Python, I was happy to see they offered a brand new workshop about Claude Code and as of this writing they are offering it again.

The course - Claude Code for Python Developers: Hands-On Agentic Coding Course. A bit wordy, but an amazing two days.

Spoiler¶

As a spoiler, my review of the course is on the main course page.

"Learning about how Claude Code works was great. Working with things like Skills, learning a workflow that functions, was what I was hoping to learn about. All of those were covered."

"I feel more comfortable with the tool itself and how to implement a basic workflow for myself with ideas on how to extend it to a whole team."

"This is one of the best training sessions I've joined in the last year across multiple platforms."

— Andrew Wegner, VP Product at Zayo

Course¶

As you can see from above, I thought this course was great. It was a two day (over a weekend) live course with about 100 participants from around the world. Based on the chat during the course, the skill set of participants ranged from "new to Claude code but experienced developer" to "somewhat familiar with Claude Code and looking to do more with it". It was a nice mix of participants. The instructor, Philipp Acsany, did a good job of answering questions, sharing content, and ensuring everyone was able to follow along. This was a very hands on workshop.

Before the course began, set up instructions were sent out. I can not overstate how much this was appreciated, because that meant the course could assume that everyone has a functioning environment to work in. This saved so much time on basic questions and allowed the course to start with the interesting content, not a tutorial on how to install Claude Code, GitHub CLI, Python and uv, Git and Zoom.

Day 1 started with the very basics of Claude. Normally, I'd be annoyed by starting with such a basic concept, but Philipp kept it engaging and more importantly, showed some best practices using Claude to scaffold a new project that I hadn't seen. Thinking that this bodes well for the rest of the course, I eagerly followed along.

As Day 1 moved on, we built upon our scaffold to develop a small application, learning how to manipulate various aspects of claude, setting up prompts and skills to assist our workflow and learning how to debug when something doesn't work. Day 1 concluded with a little bit of homework. After 4 hours in the workshop, I felt pretty confident that I could accomplish this and was happy to see that confidence was justified. After about an hour more of individual work, I completed the tasks and was ready for day 2.

Day two built on top of the homework by adding in additional features, ensuring we were able to utilize various skills, and learning more about how Claude operates under the hood. The session concluded with quick demos of additional aspects of Claude - hooks, MCPs, and Agents. Honestly, this was my biggest disappointment in the course, because it showed so many things we wouldn't be getting to, but it did fill my "research later" queue.

Conclusion¶

This isn't a cheap course. But, it was worth it to me. I approached this with two goals in mind:

Learn something for myself that I could take and apply to personal projects
Learn something for my teams so that we could use it as inspiration to make further improvements to our workflows

Both of these were met in Day 1. That made Day 2 even more fun for me, because I showed up wanting to learn more, cover more, do more. As I said in my review on Real Python: This is one of the best training sessions I've joined in the last year across multiple platforms.

Through out the course, student questions were responded to - both live via Philipp and in chat via one of Philipp's partners or from other participants. I found this aspect of the course really valuable too. I did get a handful of questions answered during it, but I was able to provide answers as well.

So, why did I only give this a 9 out of 10? What's preventing that last star?

There are aspects of the course that were touched on so briefly that I think would have been useful to dive into. These are the topics that have ended up in my research queue - hooks, mcp services, agents, agent teams. I think these could have filled another 4 hour block, but this was a weekend course and didn't fit. I'll be watching for a session that covers these topics.

The other thing - the cost can be prohibitive to students. There is the cost of the course itself: $800 normally, $500 on sale when I joined. Plus the cost of Claude. It is recommended to get at least the Max plan which runs $100 per month. I agree with that. I think if I'd only gone with Pro, I'd have hit usage limits during the course.

That said, if you can afford it (or get work to cover it as training), this is worth it for both an introduction to Claude Code and to learn about features you likely aren't using to their full power. Even with this course, I don't think I am doing that yet, but I know what to research now to get better for personal usage and for team improvements.

I Proved AI Could Beat Our Technical Interviews. Then I Had to Fix Them.

2026-03-25T15:00:00-05:00

The interview process I used to run¶

For years, I used timed online assessments as part of my hiring process. At PacketFabric, when we needed to scale the engineering team rapidly during our Series B, I helped evaluate several assessment platforms to find one that balanced candidate experience with the quality of signal we received as hiring managers. We landed on Woven, and it worked. The assessments helped us move quickly through a high volume of candidates and gave us comparable data points across applicants.

I had doubts, though. I've been opposed to LeetCode style interviews forever. Woven wasn't LeetCode as it provided amazing feedback to the candidate on what did and did not work. I liked the company so much that I joined Woven for a while. But even with the best assessment platform I'd found, I saw cracks. The candidates who passed the assessments and joined the team didn't always succeed in the ways I expected. Some produced clean code under time pressure but struggled to collaborate with the team. Others passed technical screens convincingly but couldn't translate that skill into shipping features in a real codebase with real constraints. The assessment told me they could solve a problem in isolation. It told me very little about whether they could do the actual job.

Then ChatGPT arrived, and my doubts became certainties.

In late 2022 and early 2023, I ran ChatGPT through technical assessments on LeetCode, TestGorilla, CodeSignal, Codility, HackerRank, and CoderByte. It passed most of them. Not marginally; it produced solutions that would have advanced through our pipeline. That wasn't a theoretical concern anymore. If a freely available tool could pass our technical assessment, we weren't evaluating engineering skill. We were evaluating test-taking ability, and we now had a machine that was better at test-taking than many humans.

That was 2022 and 2023. We are now several AI model generations beyond that early ChatGPT. Tools like Claude Code, Codex, Cursor, and GitHub Copilot are used by millions of developers daily. These aren't novelty chatbots any longer. They're integrated development environments that write, debug, and refactor production code. The gap between what AI could do when I ran those tests and what it can do today is enormous. Any assessment that was vulnerable to early ChatGPT is trivially solvable now.

I had to rebuild how I hire.

What timed assessments and take-homes actually measure¶

Timed assessments reward speed and pattern recognition under artificial pressure. That correlates poorly with the actual work. Nobody ships production code in 45 minutes with a countdown timer and no access to documentation. The skills that make someone fast on a Codility problem like memorized algorithms, familiarity with specific puzzle patterns, comfort performing under observation, are not the same skills that make someone effective on a team building real products.

Take-home projects have the opposite problem. Without time constraints or observation, there's no way to verify who actually did the work, how long it took, or what tools they used. Even before AI, take-homes had issues: they disproportionately disadvantage candidates with families or other time commitments, and they're difficult to evaluate consistently across candidates.

Both formats share a deeper flaw though. They optimize for a narrow technical signal while telling you almost nothing about how a person thinks, communicates, or works with others. I've seen candidates pass assessments convincingly and then struggle on the job because they couldn't explain their decisions to a colleague, couldn't adapt when requirements changed, or couldn't take ownership of a system beyond the specific code they wrote. The assessment didn't predict any of that. It just told me they could solve a contrived problem under contrived conditions.

I want to be fair to these tools, though. They feel rigorous and objective on the surface. That's why they're popular. The scores are comparable, the process is standardized, and it feels like you're making data-driven decisions. But the data is measuring the wrong thing.

What I look for instead¶

My core philosophy has stayed the same across companies even though the specific steps change based on the team, the roles, and the company's needs. The philosophy is this: I want to understand how someone thinks, how they communicate, and whether they take genuine ownership of their work. The specific technology matters, but it matters less than those three things.

How I evaluate depends on the level of the role.

Mid-level and senior engineers¶

For experienced candidates, I focus on past project deep-dives. I ask the candidate to describe something they built - not a rehearsed elevator pitch, but a real system they worked on. Then I poke at it.

The questions I ask depend on what the candidate brings up, and I rotate through several lines of inquiry based on what I'm hearing. If someone describes a system they architected, I'll ask about the tradeoffs they made and why. If they describe a project that clearly had rough patches, I'll ask what went wrong and dig into how they handled it. This isn't because "tell me about a failure" is a novel question, but because the follow-up conversation reveals depth. The most telling question, though, is usually some form of "what would you do differently if you built this again?" Candidates who were deeply involved in a project have learned something from it. They have opinions about what they'd change. Candidates who are exaggerating their involvement tend to stumble here because they haven't actually wrestled with the consequences of the original decisions.

About half of the candidates I interview can genuinely defend their decisions in this format. The other half can describe what they built but can't articulate why they made the choices they did, or what they learned from the experience. That's the signal. I'm not looking for perfect decisions. I'm looking for evidence that they understand systems deeply enough to reason about them, not just implement them.

For more senior and architecturally-focused roles, I lean harder on design tradeoff discussions. For mid-level engineers, I focus more on the "what went wrong" and "what would you change" angles, which are more accessible and still reveal a lot.

Junior engineers¶

Past project deep-dives work less well for junior candidates. They don't have enough experience to defend decisions they haven't had the opportunity to make yet. Asking a recent graduate to explain the architectural tradeoffs of their senior capstone project isn't a fair evaluation.

Instead, I use a small live coding exercise. Something that can be written in 5-10 lines of code, with multiple valid approaches. For example, a problem that can be solved iteratively and recursively. I ask them to write it one way and we talk through it for a minute or two. Then I ask them to write it the other way. I don't tell them in advance that I'll ask for the alternative approach. I want to see what they do naturally when the problem shifts.

The code itself is almost secondary. What I'm evaluating is the conversation. Can they reason about the difference between the two approaches? Can they articulate why one might be preferable in a given context? Do they get flustered when asked to think differently, or do they engage with the challenge? A junior engineer who writes imperfect code but can reason clearly about alternatives and ask good questions is far more promising than one who produces a clean solution but can't explain their thinking.

Leads and directors¶

For leadership roles, I shift to scenario and system design conversations. Less about code, more about organizational and architectural judgment. How would you structure a team to build a particular kind of system? How would you evaluate and prioritize a backlog of technical debt? How do you handle a situation where product and engineering disagree on timeline or scope? The signal here is similar to the engineering deep-dives. I'm listening for whether a candidate can reason through ambiguity, make a decision with incomplete information, and explain their thinking clearly. A strong candidate will ask clarifying questions, acknowledge tradeoffs, and arrive at a defensible position. A weak candidate will give a textbook answer that doesn't account for the messy reality of the scenario.

Technical product managers¶

I also interview Technical Product Managers (TPM), and the approach shares the same philosophy as the engineering deep-dives. I'll present a feature or problem that has already been solved so that I have detailed knowledge of what has worked. I ask them to walk through how they'd approach it. I'm watching how they think through the problem, what questions they ask, what tradeoffs they identify, and how they communicate their reasoning. Then during the conversation, I poke at their decisions the same way I would if I were working through the problem with a TPM on my team. Can they explain why they prioritized one approach over another? Can they articulate the user impact? Can they adjust their thinking when I introduce a constraint they hadn't considered? This fits directly into the communication signal I look for across every role which is the ability to reason through a problem, defend your position, and adapt when new information arrives.

The soft skills that predict long-term success¶

Technical skill gets someone through the interview. Soft skills determine whether they succeed on the team. I assess these through a combination of specific questions, observing behavior throughout the interview, and paying close attention to how candidates describe the systems and processes they've worked in.

Three signals have been the strongest predictors across my career:

Ownership without ego. I want someone who takes responsibility for the system, not just the lines of code they wrote. But ownership can become toxic when it turns into "this is my baby and it's always right and the user is wrong." The best engineers own the outcome for the user, not just the implementation. When something breaks, they don't deflect. But they also don't get defensive when someone suggests a different approach or when a user reports that the system isn't working the way they intended. You can often hear the difference in how candidates describe past work: do they talk about what "I built" in isolation, or do they talk about how the system served its users and how the team improved it over time?

Communication. Can they explain a technical decision to someone who isn't an engineer? Can they disagree with a colleague constructively? The interview itself is a communication exercise and I'm paying attention to how clearly they explain their past projects, how they handle follow-up questions, and whether they can adjust their level of detail based on the conversation. An engineer who can only communicate with other engineers who share their context will hit a ceiling quickly, regardless of their technical ability.

Curiosity and willingness to learn. The technology stack will change. The industry is already changing and AI has introduced so much disruption in such a short time that it is vital candidates are willing to learn and adapt. I've worked across many industries in my career, and the engineers who thrived through transitions were the ones who were genuinely energized by learning something new rather than threatened by it. In interviews, I listen for whether candidates describe their learning in terms of genuine interest or obligation. "I had to learn Kubernetes for the role" sounds very different from "I got interested in how our deployment pipeline could be more reliable, which led me to Kubernetes."

Designing interviews that AI can't shortcut¶

The ChatGPT experiment wasn't just a blog series - it fundamentally changed how I approach hiring. My current process is designed so that AI assistance is either irrelevant or immediately visible.

You can't have ChatGPT defend your past project decisions in a live conversation. When I ask a candidate what they'd do differently about a system they built two years ago, there's no prompt that generates an authentic answer. The deep-dive format is inherently AI-resistant because it's evaluating lived experience, not producible knowledge.

For the junior coding exercises, the code portion could theoretically be AI-assisted. But the follow-up conversation, "now write it the other way, and explain why you'd choose one approach over the other", reveals immediately whether the candidate understands what they wrote. I've caught candidates who clearly used AI for their solution and then could not explain their own code when I asked follow-up questions. The conversation is the evaluation, not the code.

My position on AI use in interviews is nuanced. I don't ban AI tools, but I do expect candidates to disclose if they're using them. If a candidate uses AI and tells me about it, we can have a productive conversation about how they used it and what their own contribution was. If we've asked, and they haven't disclosed, that's a failure of integrity that outweighs technical competence. When you're hiring someone to join a team, trust matters as much as skill.

This isn't a hypothetical concern. I've encountered candidates who were clearly using AI during interviews and couldn't explain their own answers. I'm not alone. CNBC reported in 2025 that tools like Interview Coder and Cluely now use invisible screen overlays that are undetectable by standard screen sharing, and hiring managers describe the telltale pattern of a pause, an "Hmm," and then a suspiciously perfect answer. An analysis of over 19,000 interviews by Fabric found that cheating adoption more than doubled from 15% to 35% between June and December 2025, with technical roles showing a 48% cheating rate.

I've also encountered the broader trend of candidates working at multiple companies simultaneously. This is a real and growing problem in remote engineering. A ResumeBuilder.com survey found that 37% of remote workers hold two full-time jobs, with AI tools making it increasingly feasible to juggle multiple roles within a standard workweek. Both of these trends make it even more critical that your interview process evaluates the actual human sitting across from you, not just the output they produce.

What I haven't solved yet¶

No hiring process is perfect, and I don't want to pretend mine is. There are areas I'm still actively working to improve.

Evaluating candidates from very different technology stacks remains a challenge. When someone's entire career has been in a language or ecosystem that's different from what we use, the past project deep-dive still works for assessing how they think, but it's harder to gauge how quickly they'll become productive in a new environment. I haven't found a clean solution for this that doesn't fall back on the kind of generalized assessments I've moved away from.

Scaling the process is another challenge. Deep-dive conversations take time and they're significantly more time-intensive per candidate than a timed assessment that can be administered asynchronously. When you're hiring for many roles simultaneously, the time cost is real. I've mitigated this by involving more of the team in interviews, but that creates its own overhead.

And the overall hiring pipeline is still slower than I'd like. The industry-wide process of sourcing, screening, interviewing, and closing candidates has friction at every stage. Improving my interview process doesn't fix the weeks lost to scheduling coordination, offer negotiations, and notice periods.

What actually matters¶

I've been hiring engineers across industries for nearly two decades and for companies ranging from pre-seed startups to Fortune 500 corporations. I've been wrong about candidates in both directions; people I was confident about who didn't work out, and people I had reservations about who became some of the strongest members of the team.

What I've learned is that the interview process should evaluate what actually matters on the job: Can this person reason about complex systems? Can they communicate clearly and work constructively with others? Do they take ownership of outcomes, not just code? Are they honest about what they know and what they don't?

No timed assessment answers those questions. A conversation does.

The Remote Engineering Leadership Playbook: Why Distance Doesn't Diminish Performance

2025-08-22T09:00:00-05:00

Remote Engineering Works When Done Right¶

When the U.S. Bureau of Labor Statistics released its comprehensive analysis of remote work productivity in 2024, something stood out. A one percentage-point increase in the percentage of remote workers is associated with a 0.08 percentage-point increase in total factor productivity. This covered over 60 industries from 2019-2023 (pre and post pandemic).

Rephrasing that slightly, as the number of remote workers increase at a company, the total productivity increases at the company.

For engineering leaders, this validates what many of us have seen firsthand. Remote engineering teams can outperform their co-located counterparts when proper processes exist. The difference between success and struggle lies in having the right systems in place.

After scaling multiple engineering teams, I've learned that distance becomes a liability only when we lack intentional systems. The companies thriving with remote engineering teams have built deliberate frameworks for collaboration across timezones, asynchronous communication, and meaningful ways of measuring the team's results.

Remote Engineering Success¶

Cross-Timezone Collaboration That Actually Works¶

The biggest mistake I see engineering leaders make is treating timezone differences as a scheduling problem when they should approach it as a workflow design challenge. With 80% of software engineers working at least partially remote by the end of 2025, timezone optimization becomes a core competency for leaders.

The Follow-the-Sun Development Model

Instead of forcing everyone into the same working hours, successful teams design workflows around natural handoffs. The key is documentation driven handoffs. Each transition requires clear context about what was accomplished, explicit next steps and acceptance criteria, identification of blockers and dependencies, and quality gates that prevent broken work from moving forward.

Strategic Overlap Windows

While asynchronous work drives the majority of productivity, you still need dedicated synchronous time for problem-solving and relationship building. I've found that scheduling dedicated overlapping windows a couple times per week provides face-to-face interaction without constraining individual productivity. Teams need at least one session that focuses on technical discussions and a second that builds team culture. Be intentional about these meetings though. Too many and you fall into the trap of making this a scheduling problem. Too few, and your team starts siloing itself.

Remote teams need fewer but more intentional synchronous touchpoints instead of trying to replicate in-office meeting cadences.

Asynchronous Communication as a Strategic Advantage¶

GitLab's engineering team scaled from 100 to 280 engineers in 1.5 years while remaining fully remote in the early 2020s. They deliberately avoided making their primary productivity metric, Merge Request Rate, an individual metric because they wanted to encourage collaborative behavior rather than siloed competition.

The takeaway from this, to me, is that successful remote teams optimize for collective intelligence rather than individual heroics.

Documentation-First Decision Making

Every architectural decision, technical specification, and process change should be documented before it's discussed. Writing forces precision in thinking while enabling asynchronous input from team members across timezones. This approach creates institutional memory that survives role changes and allows ideas to be evaluated on merit rather than presentation skills.

Structured Communication Protocols

Remote teams need explicit protocols for different types of information. Urgent blockers require immediate Slack notifications with clear context. Technical discussions benefit from a structured request for comments process. Status updates should follow standardized formats that enable quick scanning. Relationship building happens through dedicated informal channels and virtual chats.

The goal is more effective communication that respects individual deep work time while enabling collective progress, instead of simply increasing communication volume.

Meaningful Metrics That Drive Performance¶

Most remote engineering initiatives fail because they measure activity instead of impact. The DORA metrics provide a research-backed framework for measuring what actually matters.

Deployment Frequency measures how often your team ships code to production. Lead Time for Changes tracks the time from commit to deployment. Mean Time to Recovery shows how quickly you recover from production issues. Change Failure Rate calculates the percentage of deployments that cause problems.

These metrics work particularly well for remote teams because they focus on outcomes rather than presence. A team that deploys multiple times per day with low failure rates is performing well, regardless of whether they're working from an office or their kitchen table.

Team-Level Indicators That Matter

Beyond DORA metrics, successful remote engineering teams track cross-team collaboration frequency to prevent silos, knowledge sharing effectiveness measured by new team member productivity timelines, sprint predictability showing whether teams can reliably estimate and deliver, and technical debt management indicating whether the team builds for today or tomorrow.

The key is making these metrics visible and actionable. GitLab makes their engineering productivity metrics publicly available, creating accountability and continuous improvement.

Culture at Scale¶

Technology and processes enable remote work, but culture makes it sustainable. The most successful distributed engineering teams I've worked with share common cultural characteristics.

Ownership Over Oversight

Remote work amplifies individual accountability. When you can't rely on hallway conversations and office presence, team members must take genuine ownership of their commitments. This requires hiring for self-direction and creating systems that support autonomous decision-making.

Mentorship Through Documentation

Traditional mentorship happens as junior developers overhear senior conversations and absorb knowledge informally. Remote teams must be intentional about knowledge transfer through code reviews, architectural decision records, and documented retrospectives.

Celebration and Recognition

Remote teams need to work harder at celebrating wins and recognizing contributions. Without spontaneous high-fives and impromptu team lunches, recognition becomes a deliberate practice that requires systems and consistency.

The Competitive Advantage of Getting This Right¶

The Bureau of Labor Statistics data reveals something else too. A 1 percentage-point increase in the percentage of remote workers is associated with a 0.4 percentage-point decrease in growth in unit office building costs. This extends beyond productivity to sustainable economics, and makes me raise an eyebrow at the number of return of office mandates we have been seeing.

Companies that master remote engineering practices can access talent from anywhere rather than just expensive tech hubs, reduce operational overhead without sacrificing performance, build more resilient teams that aren't dependent on physical proximity, and create competitive advantages through superior asynchronous collaboration.

The question has shifted from whether remote engineering teams can work to whether your organization will develop the processes and culture to unlock this advantage. Many companies continue treating remote work as a necessary accommodation rather than recognizing it as a strategic opportunity.

Remote engineering success requires intentional design rather than hoping things work out naturally. For leaders willing to invest in the right frameworks, the returns in productivity, talent access, and competitive positioning are substantial and measurable.

Junior Engineer Crisis: How AI Code Generation is Reshaping Engineering Teams

2025-08-18T09:00:00-05:00

The Data That Changes Everything¶

The software development industry is experiencing a shift that's happening faster than most organizations realize. Three statistics captured my attention this year, and collectively, they paint a picture of an industry in transition:

AI now generates 41% of all code, with 256 billion lines written in 2024 alone, according to Stability AI CEO Emad Mostaque.
Software engineer job listings are down 35% from five years ago, according to The Pragmatic Engineer's analysis
AI tools make experienced, open source, software engineers 19% slower, not faster, based on a randomized controlled trial by METR Research

That last statistic flies in the face of all stories we're seeing at this point in 2025. Through my time scaling engineering teams, I've learned to pay attention when data contradicts conventional wisdom. If AI is making experienced engineers slower, what does that mean for how we think about productivity, learning, and career development in software engineering?

The implications extend far beyond individual productivity. They touch the core of how engineering teams function, how knowledge transfers between generations of software engineers, and ultimately, how we build the technical leaders of tomorrow.

The Productivity Paradox¶

The conventional narrative around AI in software development has focused on speed and efficiency. AI improves employee productivity by up to 66% across various roles, according to recent studies. Yet when it comes to experienced software engineers, the opposite appears to be true.

The METR Research study revealed that when experienced engineers used AI tools, they took longer to complete tasks compared to working without AI assistance. This finding challenges the central assumption driving AI adoption in engineering teams.

Why does this happen?¶

In my experience, software development isn't just about code generation. It's about problem-solving, architecture decisions, and understanding complex system interactions. When software engineers rely on AI to generate code, they often spend additional time not doing these. Instead they are reviewing and understanding what the AI spit out to ensure it aligns with system requirements. Then they spend time debugging the code due to missed requirements, edge cases, or lack of context provided to the coding assistant.

This productivity paradox reveals something crucial. The value of software development isn't in typing speed. It's in thinking speed.

Reality Check¶

Meanwhile, almost all companies are investing in AI, but just 1% believe they are at maturity, according to McKinsey's 2025 workplace analysis. This suggests that even organizations themselves are struggling to effectively integrate AI tools into their development processes.

The disconnect between AI's promise and its current reality in software development creates a unique opportunity for engineering leaders who understand how to navigate this transition thoughtfully.

The Mentorship Gap¶

Traditional software engineering career progression has followed a predictable pattern:

Junior Engineer -> Mid-level Engineer -> Senior Engineer -> Technical Lead

This progression relied heavily on mentorship, code reviews, and the gradual transfer of knowledge from experienced engineers to new team members. AI is disrupting this learning pipeline in ways we're just beginning to understand.

The Traditional Knowledge Transfer Model¶

Historically, junior engineers learned through writing code that was reviewed by senior engineers. They also participated in pair programming and code reviews that taught best practices and patterns, and working on increasingly complex tasks. Crucially, they also debugged their own mistakes and learned from failure. Generative AI deprives the youngest team members of this skill because they are debugging code that was generated for them rather than by them. They don't have the depth of knowledge over this code snippet. They haven't spent hours working to solve a problem in this one line of code. The failure of the code is not their own, it's the AI's.

Each of these steps built not just coding skills, but engineering judgment to teach them the ability to make good technical decisions under uncertainty.

The AI-Disrupted Model¶

Now, increasingly, the model is that the AI generates code based on natural language specifications. Then a senior, not junior, engineer reviews the output. As the debugging loop starts, the focus is on the AI prompt engineering instead of the code. Code reviews become a validation of the AI and aren't teaching moments any longer. Junior engineers solve less problems and babysit the AI instead.

This shift significantly changes the nature of mentorship on engineering teams.

The Hidden Cost¶

When senior engineers spend their time reviewing AI-generated code instead of mentoring junior engineers, we lose the critical element of human knowledge transfer.

A senior engineer reviewing a junior's code can ask questions like: - "Why did you choose this approach?" - "What other options did you consider?" - "How would this handle increased load?" - "What happens if this service is unavailable?"

These questions build engineering judgment. But when reviewing AI-generated code, the questions become: - "Did the AI choose the right pattern?" - "Does this handle our edge cases?" - "Is this consistent with our architecture?"

The learning opportunity shifts from building problem-solving skills to validating AI decisions.

What We're Really Losing¶

The impact of AI on junior engineering roles goes beyond individual career paths. We're potentially losing institutional knowledge, problem-solving capabilities, and the human intuition that comes from learning to code through struggle and iteration.

Problem Decomposition Skills¶

One of the most valuable skills a software engineer develops is the ability to break complex problems into smaller, manageable pieces. This skill typically develops through experience. This is through encountering problems that are too big to solve all at once and learning to approach them systematically.

When AI handles this decomposition automatically, junior engineers don't develop this critical thinking muscle. They become skilled at describing problems to AI rather than solving them independently.

Debugging Intuition¶

Experienced engineers often talk about having a "gut feeling" about where bugs might be hiding or what might cause a system to fail under load. This intuition develops through years of debugging their own mistakes and understanding how systems fail in practice.

AI-generated code fails differently than human-written code. It might be syntactically correct but miss business logic edge cases. It might follow patterns perfectly but make assumptions about data that don't hold in production. Learning to debug AI code is a different skill from learning to debug human reasoning errors.

Architectural Thinking¶

Understanding why certain architectural patterns exist, when to apply them, and how they impact system behavior requires experience with the consequences of different choices. This understanding traditionally developed through making mistakes, seeing systems break, and learning from the aftermath.

When AI makes many of these architectural decisions automatically, junior engineers may learn to recognize good patterns without understanding why they're good or when they might be inappropriate.

The Compound Effect¶

Perhaps most concerning is the compound effect of these changes. If junior engineers don't develop problem-solving skills, debugging intuition, and architectural thinking, who becomes our next generation of senior engineers?

Software engineers are finding it harder to get jobs in 2025 due to changing hiring standards, according to analysis from the software engineering community. The bar for what constitutes "junior engineer" skills is rising, but the pathways to develop those skills are being disrupted by AI.

The Strategic Response¶

The organizations that thrive in this transition won't be the ones that embrace or reject AI wholesale. They'll be the ones that thoughtfully integrate AI while preserving the human elements that create strong engineering teams.

Understanding the Market Reality¶

Despite these challenges, employment opportunities for software engineers are still expected to grow by 20%, according to recent market analysis. This suggests that demand for engineering talent remains strong, but the nature of that talent is evolving.

McKinsey's analysis indicates that AI has the potential to fundamentally transform software development processes, but successful transformation requires deliberate strategy, not just tool adoption. This point bears repeating because it's vital for leaders to understand.

Attention

Successful transformation requires deliberate strategy, not just tool adoption

Bringing Copilot, Cursor, Claude Code or other AI coding assistants to your team does not guarantee success. It is a step in being successful in the AI transformation. It is not the step.

Three Strategic Approaches¶

Based on my experience scaling engineering teams and observing successful AI integration, three strategic approaches are emerging.

The first approach is redefining what it means to be a "Junior Engineer." Instead of eliminating junior positions, successful organizations are redefining what junior software engineers need to be good at. Traditionally, a junior engineer would be able to write syntactically correct code, follow team development patterns, understand the language and framework used by the team to a degree, and be able to implement requirements that are well defined.

However, a new AI-era junior engineer needs different skills. They need to be able to analyze and decompose a problem, showing system thinking and architecture understanding. This enables them to collaborate with AI coding assistants to generate solutions that actually solve the problem. Critically, they can perform a review of this output.

The most effective teams I've observed use AI as a teaching tool rather than a replacement for learning. Junior software engineers work with both AI tools and senior engineers, using AI to handle boilerplate while focusing human mentorship on architectural decisions and problem-solving approaches. This keeps several of the mentorship gaps minimal and builds a foundation that today's junior engineers can build on as they grow in their careers.

I've also heard of teams implementing regular "AI-free" coding sessions for engineers to ensure they can solve problems without the assistance of the AI tooling to build their early troubleshooting and debugging muscles. This type of thing also has been extended to code reviews, where engineers must be able to explain why the AI tool took certain approaches and if there are alternatives that might have been better.

Personally, these approaches feel somewhat academic. A junior engineer is still a professional not a student in school. I think there could be good training around these ideas, but I don't think it should feel like we are taking away a tool. Instead, teach how to use it correctly while filling in the knowledge gaps.

In my experience, successful teams are making knowledge transfer deliberate through recording architectural decisions that were made and what alternatives were considered. Through troubleshooting sessions, problem solving sessions, and "pair support" to dive into complex system problems.

Mentoring from anyone to anyone is important too. Everyone on the team has something to contribute and teach. Whether it's complex system architecture that a senior engineer shares with a more junior team member or having a junior engineer lead a session on how they solved a problem. All of this is important for the whole team.

Redefining Junior Engineer Value¶

The key insight is that AI doesn't eliminate the need for junior developers. It changes what makes junior software engineers valuable, though.

Code Generation to Code Curation¶

In an AI-first world, junior software engineers become code curators rather than code generators. They will evaluate AI generated solutions for correctness, efficiency and maintainability. They'll identify edge cases the AI misses. They'll take AI generated code and ensure it works within their area of responsibility.

New Core Competencies¶

The most successful junior software engineers I've worked with in AI-enabled teams demonstrate the ability to think about entire systems, assess code quality, break down problems clearly, and strive to learn and adapt to new tools which allows them to build a method of debugging code they didn't write themselves.

Looking Forward¶

The software engineering industry is experiencing a shift, but it's not the first time. We've navigated the transition from assembly language to high-level languages, from procedural to object-oriented programming, from desktop to web to mobile development. Each transition created new opportunities for those who adapted thoughtfully.

The current AI transition is no different, but it requires us to think carefully about what we're optimizing for. If we optimize purely for short-term code generation speed, we risk creating a future where we have powerful AI tools but fewer humans who understand how to use them effectively.

The Organizations That Will Win¶

The organizations that thrive in this transition will be those that preserve human judgment while leveraging AI capabilities. Organizations that invest in developing people while adopting new tools and capabilities will have a key success factor. These are the organizations that create space for engineers to learn and build engineering thinking into their processes. Most importantly, teams that maintain their cultural values while adapting processes will find engagement instead of resistance.

The Individual Path Forward¶

For individual engineers, especially those early in their careers, the path forward involves:

Embracing AI as a tool while building problem-solving skills that transcend any specific technology.
Focusing on system thinking and architectural understanding that AI currently cannot replicate.
Developing communication skills that allow you to work effectively with both AI tools and human teams.
Building debugging and quality assessment capabilities that work regardless of who or what generated the code.
Maintaining curiosity about how things work, not just how to make them work.

Conclusion¶

The junior engineer crisis isn't really about AI replacing entry-level engineers. It's about ensuring that as we integrate powerful new tools into our development processes, we don't lose the human elements that create strong engineering teams and effective technical leaders.

I argue that every significant technology shift creates winners and losers. The winners are those who adapt early and thoughtfully, who understand both the capabilities and limitations of new tools, and who invest in building the human skills that remain uniquely valuable.

The current moment represents a unique opportunity for engineering leaders to shape how AI integration happens in their organizations. The choices we make now about hiring, training, mentorship, and team structure will determine whether AI makes our engineering teams stronger or simply faster.

AI is already reshaping our industry. The question is whether we'll guide it in directions that build stronger teams and better engineers, or whether we'll optimize for short-term productivity at the expense of long-term capability.

The junior software engineers we hire and train today will become the senior engineers leading teams in 2030. How we prepare them for that role, in partnership with AI rather than in replacement by it, may be one of the most important strategic decisions we make as engineering leaders.

Please join the conversation over on LinkedIn. I've split this article across three posts over there and would love to hear your feedback

Python Gotcha: Reusing Generators Returns Nothing

2025-07-22T09:00:00-05:00

Introduction¶

In the previous article, we looked at logging uncaught exceptions. Let's utilize the log output from that post for another common task: error log file processing. This example is going to be pretty simple, as the error log for the post is tiny, but in a production environment this could be hundreds of thousands of lines, or gigabytes in size. Potentially, that's a lot to shove into memory for processing. But that's where a generator can come in to help.

Generators do not store their results, instead they maintain state and yield the result back to the caller. This means each line in a log can be processed and returned, without loading the entire file.

Flashback¶

As a reminder from the previous article, the log file being used looks like this:

2025-07-14 22:30:44,061 __main__ INFO Application start
2025-07-14 22:30:44,061 __main__ CRITICAL uncaught exception, application will terminate.
Traceback (most recent call last):
File "/home/andy/main.py", line 31, in <module>
    main()
    ~~~~^^
File "/home/andy/main.py", line 27, in main
    logger.info(divide(a,b))
                ~~~~~~^^^^^
File "/home/andy/main.py", line 21, in divide
    return a/b
        ~^~
ZeroDivisionError: division by zero

There is one INFO entry and one CRITICAL entry.

yield vs return¶

It's important to understand what makes a generator different from a regular function. The key distinction is the yield keyword. When a function contains yield, Python treats it as a generator function, which behaves differently from functions that use return.

A regular function with return executes completely, returns back a single result and then terminates. A function with yield creates a generator object that can pause execution, return a value, and later resume from exactly where it left off. This is what enables the memory-efficient and lazy evaluation that makes generators powerful.

Gotcha¶

Log processing¶

A generator result can only be utilized one time. Whether you are using a generator to output the next item in a sequence or process a file line by line, once you have passed an iterable or exhausted the generator, it doesn't get reused. This simple generator demonstrates the issue.

def read_log_lines(filename):
    with open(filename, 'r') as f:
        for line in f:
            if 'CRITICAL' in line:
                yield line.strip()

error_logs = read_log_lines('app.log')

error_count = len(list(error_logs))
print(f"Found {error_count} CRITICAL lines")

recent_errors = [log for log in error_logs if '2025' in log]
print(f"Recent errors: {len(recent_errors)}")

At first glance, it looks like this will read the log file, count the number of errors and then output how many of those were in 2025 (or contain the string 2025). However, the actual output is different.

Found 1 CRITICAL lines
Recent errors: 0

error_logs is a generator object. If a function yields, it is a generator. As error_count is initialized, it processes the error log and yields back any critical lines. The list() function will consume the entire generator (file). A few lines later, the developer wants to see how many of these are recent errors and attempts to go through the error_logs generator again. Success! No recent errors!

Right?

No, and looking at the log quickly shows that.

Fibonacci¶

Let's use a generator to build the Fibonacci sequence. Spoiler for interviews! In this case, I'm going to use a generator to get the first 10 items. Then print out the first 5 and then try to print the entire list of 10 items.

def fibonacci(n):
    a, b = 0, 1
    for _ in range(n):
        yield a
        a, b = b, a + b

fib_numbers = fibonacci(10)

print("First 5 numbers:")
for i, num in enumerate(fib_numbers):
    print(num)
    if i >= 4:
        break

print("Entire List:")
full_list = list(fib_numbers)
print(full_list)

The output for this is:

First 5 numbers:
0
1
1
2
3
Entire List:
[5, 8, 13, 21, 34]

Notice that the full_list variable only contains the items remaining on the generator. Since the first 5 (indexes 0 through 4) were printed, they are no longer part of the generator. When the full list is printed, only the remaining items can be printed.

Solution¶

The solution to the problem is easy enough. Call the generator function again. For example, with the log code from above:

error_logs = read_log_lines('app.log')
error_count = len(list(error_logs))
...
error_logs = read_log_lines('app.log')  # Call again and create a new generator
recent_errors = [log for log in error_logs if '2025' in log]

For the Fibonacci code, you would call fib_numbers = fibonacci(10) again before printing the full list.

Obviously, there is a down side here with duplicate processing of the same data due to running the generator twice. This could probably be solved with some logic adjustments to the generator or the code calling the generator, but that'll vary by application depending on what the generator is doing.

Conclusion¶

The important thing to take away from this is that once you have iterated over an item in a generator, it's no longer part of a generator. This means that if you want to get clever and see if there are more items in a generator, or determine the next item, you've consumed the next item.

The power of generators, especially when processing large amounts of data, can't be understated. But, at the same time, it's important to know that reusing an exhausted generator or attempting to access a previous item directly from the generator is not going to work. Instead, to reuse generator logic, call the generator function again to create a new generator object, or convert to a list if memory permits and multiple iterations are needed.

Python Gotcha: Logging an uncaught exception

2025-07-14T23:00:00-05:00

Introduction¶

A well built application will use logging instead of print statements. An exceptionally well built one will log in such a way that additional context is added to each log message and be consumable by a log aggregation service. Perhaps I'll write up such an article in the future. For now though, let's focus on a single problem.

Here is some sample code to demonstrate the problem.

import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

handler = logging.FileHandler("app.log")
formatter = logging.Formatter("%(asctime)s %(name)s %(levelname)s %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)

def divide(a, b):
    return a/b

def main():
    logger.info("Application start")
    a = 10
    b = 0
    logger.info(divide(a,b))
    logger.info("Application end")

if __name__ == "__main__":
    main()

Briefly, this sets up the app.log file to receive our log messages. The main function is going to divide two numbers, log the result, and end the program. Pretty simple.

Except, in this case, it is dividing by zero. This throws an error and crashes the program.

The console spits out a stack trace

Traceback (most recent call last):
File "/home/andy/main.py", line 22, in <module>
    main()
    ~~~~^^
File "/home/andy/main.py", line 18, in main
    logger.info(divide(a,b))
                ~~~~~~^^^^^
File "/home/andy/main.py", line 12, in divide
    return a/b
        ~^~
ZeroDivisionError: division by zero

The app.log file contains a single line:

2025-07-14 22:20:04,551 __main__ INFO Application start

Gotcha¶

Where is the gotcha here? The stack trace is right there!

You are, of course, right. However, imagine that this was not a simple application, but instead a production application that sends logs to a central service. Your application crashed and no one was watching the console. Your app.log file has no information. It says the application started and then...nothing. What happened? Is it still running?

As you dig through running processes, or check a /health end point for responses, you find out that it isn't running. That took a lot of time, and production isn't responding.

You've lost all visibility to what happened in your application at the most critical moment. When it crashed and spit out a stack trace, you want as much detail as you can get.

Solution¶

The solution is sys.excepthook. This is called when any exception is raised and uncaught, except for SystemExit. It's pretty easy to utilize as well. A few small changes to the above code will allow us to log this completely unexpected ZeroDivisionError.

import logging
import sys

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

handler = logging.FileHandler("app.log")
formatter = logging.Formatter("%(asctime)s %(name)s %(levelname)s %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)

def handle_uncaught_exception(exc_type, exc_value, exc_traceback):
    logger.critical(
        "uncaught exception, application will terminate.",
        exc_info=(exc_type, exc_value, exc_traceback),
    )
sys.excepthook = handle_uncaught_exception


def divide(a, b):
    return a/b

def main():
    logger.info("Application start")
    a = 10
    b = 0
    logger.info(divide(a,b))
    logger.info("Application end")

if __name__ == "__main__":
    main()

The important bit is the new handle_uncaught_exception function and the sys.excepthook line (with appropriate import statement).

Someone running this in the console will notice that there is not a stack trace dumped to the console now. Instead, our log contains important information:

2025-07-14 22:30:44,061 __main__ INFO Application start
2025-07-14 22:30:44,061 __main__ CRITICAL uncaught exception, application will terminate.
Traceback (most recent call last):
File "/home/andy/main.py", line 31, in <module>
    main()
    ~~~~^^
File "/home/andy/main.py", line 27, in main
    logger.info(divide(a,b))
                ~~~~~~^^^^^
File "/home/andy/main.py", line 21, in divide
    return a/b
        ~^~
ZeroDivisionError: division by zero

This is great! Now when troubleshooting this failing application and looking at the logs, we can easily see that an exception occurred. Additionally, with proper updates to the logging, more context can be provided such as the values of a and b. While it's easy enough to figure out that b is 0 in this simple example, the context in a larger production application could save a ton of troubleshooting time.

Conclusion¶

Logging is vital to knowing what your application is doing. But, it's even more important to determining why it stopped working. If your logs aren't providing information when the application crashes, it's functionally useless. By implementing an excepthook, you can catch and properly log uncaught exceptions.

I know some of you are coming up with alternatives. Terrible ideas like wrapping the entire main block in a try/except. There are legitimate reasons to throw an exception. In this case, a ZeroDivisionError is a great exception to catch. But, you'd want to do it around as small of a code block as possible.

This is a clean way to catch truly unexpected exceptions, not something a developer could have anticipated and, perhaps, fixed with additional input validation.

How to fix GitLab 18 error regarding git_data_dirs

2025-07-06T08:00:00-05:00

Introduction¶

Several years ago I set up GitLab in my home environment. I appreciate past me documenting the basic steps I took to do this, because I'll need them again at some point when I make upgrades to my home lab. I've documented some things I've done (like setting up GitLab runners, or dealing with Grafana being deprecated within GitLab or utilizing GitLab to automatically backup my Obsidian notes) Unfortunately, I didn't document everything. One of those things that I didn't document was changing where my repositories are stored on disk by default.

In GitLab 17.8 (January 2025), a new deprecation warning started appearing.

git_data_dirs has been deprecated since 17.8 and will be removed in 18.0. See https://docs.gitlab.com/omnibus/settings/configuration.html#migrating-from-git_data_dirs for migration instructions.

I'm a little annoyed that there were only 5 months of notice on this because major versions are released in May. In any case, if you attempt to update to version 18 or beyond and are utilizing git_data_dirs, the upgrade will fail.

Solution¶

As usual, GitLab is good at providing documentation on resolving and migrating through the deprecations. However, on my first attempt I encountered an error. I believe it's because I missed a note buried in the text - not the code blocks - the first time.

Note that the /repositories suffix must be appended to the path because it was previously appended internally.

The solution is to open /etc/gitlab/gitlab.rb (you probably should make a backup first). Find the git_data_dirs line. For me this was around line 455. Then comment out this entire block.

Then find (or add) the gitaly['configuration'] block. This was immediately after the git_data_dirs section for me. Uncomment it, and add the appropriate path (from your git_data_dirs block) and add /repositories to the end of it.

Once you are done, run

gitlab-ctl reconfigure
gitlab-ctl restart

Give it a minute to fire up all the GitLab subcomponents, and then you should be able to update GitLab beyond version 18.

New code section¶

My gitlab.rb now has this:

#git_data_dirs({
#  "default" => {
#    "path" => "/previous/path/to/repos"
#   }
#})

### Gitaly settings
gitaly['configuration'] = {
storage: [
    {
    name: 'default',
    path: '/previous/path/to/repos/repositories',
    },
],
}

Conclusion¶

With this quick change, I can continue to hold the repositories at a location of my choosing. The biggest thing is to add /repositories to the path in the new Gitaly configuration. With this change, I can continue to utilize the current version of GitLab - a tool that I still find invaluable.

Python Gotcha: Identity vs Equality - When 'is' Fails Unexpectedly

2025-06-17T08:00:00-05:00

Introduction¶

In the comparisons gotcha I wrote a few years ago, I briefly touched on is vs ==.

Put simply, is should ONLY be used if you are checking if two references refer to the same object. Remember, is compares object references.

Even more simply, is is not checking value. Let's take a look at a couple examples.

Gotcha¶

Integers¶

Python, specifically CPython, caches the values of -5 through 256 (inclusive). This means that these small integer values will always refer to the same object.

Note the phrasing there - "the same object".

Outside of that range, though, the same is not true.

>>> a = 100
>>> b = 100
>>> a is b
True
>>> a = 257
>>> b = 257
>>> a is b
False

In both of the above examples, using a == b would have returned True. The mistake was assuming that is does the same thing. It does not.

Strings¶

String interning is a method of storing only one copy of each distinct immutable string value. Immutable strings can't be changed. Not every string will be interned though. Let's take a look:

>>> a = "Hello"
>>> b = "Hello"
>>> a is b
True
>>> a = "Hello World"
>>> b = "Hello World"
>>> a is b
False
>>> a = "Hello_World"
>>> b = "Hello_World"
>>> a is b
True

The first and last example interned the strings, showing that a and b refer to the same object. But, the second example - Hello World - didn't get interned, so a and b refer to different objects. Why is this?

The short and simply answer is that any string that has only numbers, letters or underscores will be interned. Since Hello World contains a space, it would not be interned.

The Solution¶

To a new developer that has seen tutorials that read if a is True or if b is None, a conditional for integers or strings following the same pattern appears to be comparing values. If they test it with small, positive numbers or simple one word strings, the assumption holds up.

But, == is for comparing values! Each of the above examples would return True by changing the statement to a == b.

The few times that is is appropriate are when you are checking True/False or None. Otherwise, the vast majority of the time, you want to use an equality check (==)

The Python PEP8 programming recommendations state:

Comparisons to singletons like None should always be done with is or is not, never the equality operators.

The linters in the Python ecosystem report on the usage of is vs == too. flake8 has E711 - Comparison to None should be 'cond is None:'. ruff has a similar report with it's None comparison check.

I highly recommend a linter for your projects to catch this, and other problems that go against best practices.

Remember, is compares object references, not object equality

Leading Distributed Teams: Lessons from Managing Global Engineering Teams

2025-04-15T01:00:00-05:00

Introduction¶

Leading engineering teams across multiple time zones has been a defining part of my career. I have taken courses on how to manage a remote team, how to implement TeamOps (both offered by GitLab) and I've managed teams spanning half of the global time zones, with team members located across every continent except Antarctica. This global distribution brings tremendous advantages in terms of talent access and round-the-clock development capability, but it also presents unique leadership challenges that require intentional strategies.

As software development continues to embrace remote work - kind of, it has started falling out of favor since the end of the pandemic - the ability to effectively lead and scale distributed engineering teams has evolved from a nice-to-have skill to a critical competency for senior leaders. The lessons I've learned managing global teams have shaped my leadership approach and provided insights that apply across both to engineering and to the broader organization.

Building Foundations for Distributed Success¶

The foundation of any successful distributed team begins with intentional communication. Tools like Slack, Zoom (with recording capabilities), Atlassian suite (Jira/Confluence), GitHub/GitLab, and collaborative document platforms like Google Workspace aren't just conveniences. They are the essential connective tissue of distributed teams.

However, having the right tools is only the beginning. More important is establishing clear principles for how these tools should be used. In my experience, the most successful distributed teams operate with a strong bias toward asynchronous communication. This means documenting decisions thoroughly, creating detailed specifications in advance of implementation, and ensuring appropriate synchronous discussions are recorded and summarized for team members in different time zones. Things like team meetings, troubleshooting sessions, and deep dives should be recorded and shared so that others can consume this information too.

Documentation is the backbone of effective collaboration. Unlike co-located teams that can rely on impromptu conversations or whiteboard sessions to clarify questions, distributed teams need comprehensive documentation that stands on its own. This includes architectural decisions, specification documents, onboarding guides, and even meeting notes.

A study by GitLab in 2021 (during the height of the pandemic) found that less than half of respondents felt their company was sharing company goals, creating and documenting processes and standards or promoting visibility across teams well. Intentional communication is important in all work places, but even more so when everyone is located in geographically different areas of the world.

I've found that building a "document-first" culture pays enormous dividends. When team members know that the first question in response to "How do I...?" will be "Did you check the documentation?", they naturally begin to contribute to and rely on that shared knowledge base. This reduces repeat questions, accelerates onboarding, and creates a more equitable information environment across time zones.

Decision-Making Frameworks Across Time Zones¶

One of the most challenging aspects of leading distributed teams is balancing the need for inclusive decision-making with the necessity to move projects forward. Over time, I've developed a time-gated decision framework that respects input from all regions while preventing decision paralysis.

For non-emergency decisions, I post proposals to our shared channels with clear timelines for feedback. For example: "I'll be creating Jira stories based on this architectural approach starting next Monday unless concerns are raised." This provides a minimum of one business day (often more) for team members across all regions to review and comment, while also creating a clear decision point that prevents endless deliberation.

For decisions requiring input from specific individuals, I'll explicitly mention them in the initial message. This creates accountability while also signaling to others who the key stakeholders are for that particular domain.

Not all decisions can wait, of course. Production outages or customer-impacting bugs demand synchronous communication and rapid response. In these cases, available team members need to be empowered to make decisions with the information at hand, with the understanding that these decisions will be reviewed and possibly refined once more of the team is available.

The balancing act of knowing when to wait for comprehensive input versus when to move forward with available information is one of the most nuanced skills in distributed leadership. In my experience, being explicit about which type of decision is being made helps team members adjust their expectations appropriately.

Technical Infrastructure for Global Development¶

The technical infrastructure supporting distributed development deserves particular attention. Continuous Integration/Continuous Deployment (CI/CD) pipelines, which are beneficial for any engineering organization, become absolutely essential for distributed teams. Automated testing, code quality checks, and deployment processes create consistency across time zones and reduce the risk of human error. This isn't limited to just a "time zone" thing though. It creates consistency across the team. Everyone's code, from the newest member of the team to the most veteran member, has their code commits go through the exact same process.

Code review practices need careful consideration in distributed environments. When reviewers might be asleep during a developer's working hours, teams need clear expectations about review timeframes. I emphasize that while we strive for quick reviews, team members should be prepared to either continue with different work or pivot to another task if a review is delayed. As the team grows and more individuals are working similar hours these delays will reduce, but initially, it is a concern that the team members have deliberately work on so that someone isn't always delayed.

Deployment strategies must also account for global distribution. Whether your team deploys each commit directly to production or follows a more scheduled release cadence, the key is establishing clear, documented processes that anyone on the team can follow regardless of location. When setting up these processes, consider who needs to be available during deployments, how deployment failures will be handled across the team, and whether certain deployment windows should be avoided due to regional holidays or weekend coverage.

In my experience, the most successful approach is establishing deployment processes that don't require real-time coordination between team members in different time zones. This might mean investing more heavily in automated testing, feature flags, and rollback capabilities, but the resulting independence pays dividends in team efficiency and satisfaction.

A Support Model¶

One of the natural advantages of a globally distributed team is the ability to provide around-the-clock coverage without requiring anyone to work unusual hours. We've implemented a rotation support model, where team members handle support duties during their local working hours. This creates continuous coverage while respecting everyone's work-life balance. Depending on the product, this could mean that the team only provides support during business hours or it could mean that the support follows the sun. In either case, the type of support provided must be clearly communicated to both the team and to end users.

This doesn't mean team members are never contacted outside working hours. True emergencies sometimes require waking someone with specialized knowledge. However, two important principles guide these situations:

First, if someone is contacted outside working hours to address an issue, they receive compensatory time off. This creates accountability in the escalation process. People think twice before waking a colleague unless truly necessary.

Second, and perhaps more importantly, each off-hours escalation triggers a review of our documentation and training. If someone needed to be woken for a particular issue, that indicates a gap in our knowledge distribution that needs to be addressed. This transforms what could be a negative experience into an opportunity for team improvement.

Over time, this approach creates robust documentation and broader knowledge distribution across the team, reducing the frequency of off-hours disruptions and building resilience into the organization. This is important, because documentation takes time. It's rarely a one shot deal.

Evaluations in Distributed Teams¶

Evaluating performance fairly across distributed teams requires deliberate attention to measurable outcomes rather than observable behaviors. I focus primarily on deliverables. Are teams hitting their targets and delivering on time and on budget? Are we meeting the KPIs we've established for deployments, code reviews, specifications, feature delivery, and performance improvements?

For managers specifically, I evaluate whether they're maintaining regular one-on-one meetings with all team members, regardless of location, and whether they're developing clear career succession and training plans for each report. These expectations apply uniformly across the organization, creating consistency in how leadership effectiveness is measured.

To ensure that geographical distance doesn't create invisible barriers to advancement, I maintain skip-level one-on-ones with team members across all regions. This gives everyone direct access to higher-level leadership and provides me with unfiltered insights into how the team is operating and individual contributions that might otherwise be obscured by distance.

I've found that managers who excel in distributed environments share certain qualities: they communicate proactively, document thoroughly, avoid making assumptions about team members' contexts, and create equitable opportunities for visibility and recognition regardless of location. These behaviors become key elements in leadership development and promotion criteria.

Developing Talent Globally¶

Hiring for distributed teams requires evaluating candidates not just for technical skills but for their ability to thrive in a remote, asynchronous environment. Beyond the job-specific requirements, I look for clear written and verbal communication skills, as team members will interact with colleagues, leaders, and customers from diverse linguistic and cultural backgrounds. Comfort with asynchronous work patterns and self-direction is essential, as are strong documentation habits that support knowledge sharing. Cultural adaptability and awareness round out the key attributes that predict success in distributed environments.

Once hired, ensuring equitable career development opportunities across regions becomes essential. During review cycles, we systematically discuss everyone's performance and growth trajectory. I expect managers to speak knowledgeably about each team member's contributions and aspirations, regardless of where they're located.

The skip-level one-on-ones mentioned earlier serve a dual purpose here. First, they help identify talent that might otherwise be overlooked due to geographical distance from headquarters or senior leadership. These conversations also build relationships that help me understand individual motivations and career goals across the organization.

One particularly effective practice has been creating cross-regional project teams that give team members exposure to colleagues and challenges outside their immediate locale. This broadens perspectives while creating mentorship and leadership opportunities that might not arise within more regionally confined teams.

Conclusion¶

As I've moved through increasingly senior leadership roles, the principles of effective team management have become even more important to my leadership approach. The future of engineering leadership demands comfort with distributed teams. Demands. Leading a team, whether it's fully remote or hybrid, is a skill that leaders need if they want to have access to talent.

The most successful technical leaders in this environment will be those who design systems and processes that work asynchronously by default, create cultures where documentation and knowledge sharing are valued and rewarded, and balance inclusive decision-making with the need to move forward. They will build technical infrastructure that supports distributed development, implement support models that leverage global distribution, ensure performance evaluation focuses on outcomes rather than presence, and develop hiring practices that identify talent suited for distributed work.

These capabilities represent the future of engineering leadership at the highest levels. For aspiring technical leaders, developing comfort and competence in distributed team management isn't just a nice-to-have—it's increasingly essential preparation for senior technical leadership roles.

Technical Debt Management: A Strategic Approach

2025-04-08T09:00:00-05:00

Introduction¶

The term "technical debt" has become a catch-all phrase, it's used to describe anything from poorly written code to outdated systems. But at its core, technical debt represents the very real trade-off of choosing immediate delivery over long-term maintainability. For companies caught between aggressive timelines and the need for sustainable systems, managing this debt becomes more than just a technical challenge. It's a strategic imperative.

Understanding Technical Debt¶

Technical debt manifests differently at various companies. Large organizations struggle with legacy systems built decades ago, while smaller companies struggle with systems built hastily last quarter that are already creaking under unexpected scale or scope expansion.

The primary sources of technical debt during rapid scaling often emerge from several common scenarios. I've witnessed early architectural decisions made with limited information create significant challenges. When we built our first API at one startup, we made assumptions about how users would utilize the API and the portal built on top of it that proved incorrect. We made assumptions about our own product line that didn't hold true as we scaled. As our user base grew, as our product line grew, as our internal structure grew, we found that we'd built ourselves into a cage that wasn't easy to build our way out of. Acquisition integration introduces another layer of complexity, as we often need to integrate systems built with different architectural assumptions and technical standards.

Feature pressure also plays a huge role. When racing to meet competitive threats or capitalize on market opportunities, corners get cut. These aren't necessarily bad decisions at the time, but they accumulate over time. Team rotation creates knowledge gaps. As teams and companies scale, original architects and developers move into management, other roles, or to other companies taking vital context with them if proper knowledge transfer isn't prioritized.

The "move fast and break things" ethos that drives early success can quickly become a liability if technical debt isn't managed proactively. In these environments, yesterday's clever hack becomes today's bottleneck, and today's workaround becomes tomorrow's single point of failure.

The Real Cost of Technical Debt¶

Technical debt is often discussed in abstract terms, making it difficult for non-technical stakeholders to appreciate its impact. To make these costs tangible, I find it helpful to categorize them into different areas of impact.

Unmanaged technical debt progressively slows development. What once took days begins taking weeks as developers navigate around increasingly brittle code. Systems built around technical debt typically require more hands-on management. For example, in one role, we found support was seeing the same pattern of problems over and over again, despite engineering effort to solve the problem. Engineers were spending so much time maintaining these systems that new development slowed to a crawl.

Which leads to what I think is the most damaging aspect of unmanaged technical debt. Stifled innovation. When engineers spend their creative energy working around limitations rather than solving new problems, competitive advantage erodes. This cost is hardest to quantify but most devastating long-term. Technical debt becomes most visible when it impacts customers. Whether this manifests through outages, performance issues, or security vulnerabilities, it impacts customers in some way. By then, the solution is typically more expensive and disruptive than if addressed proactively.

When to Address vs. When to Defer¶

Not all technical debt requires immediate attention. The key is distinguishing between strategic debt that enables necessary velocity and toxic debt that creates compounding problems. Here's how I approach these decisions:

The Upgrade Case Study¶

As I mentioned above, at one company, we faced a critical decision about our API infrastructure. We had built an initial API quickly as a skeleton for the product, but it wasn't designed for the scale and feature set we now needed. As the team grew rapidly, we had to decide whether to focus exclusively on building a new production ready API, or gradually migrate from old to new while continuing feature development on both.

After analysis, we estimated that implementing new product features in the old API would actually take longer than building a robust new API and implementing the new features there first. The tipping point came when our team estimated that one specific new product line would take less time to deliver by building the skeleton of a new API, setting it up properly, and implementing the new feature than it would to build the feature into the current API.

The clean-cut approach allowed new team members to focus entirely on the new API, infrastructure, and features. They came up to speed quickly by building a product they knew from the ground up. We ported the old API functionality to achieve parity, added new features as an incentive for customers to migrate, and then deprecated and ultimately shut down V1.

This counterintuitive approach, where addressing technical debt actually accelerated delivery, is exactly the kind of strategic analysis companies need.

Why Early Is Cheaper¶

A fundamental principle of technical debt management is that the cost to update almost always increases with time. In my experience, moving one version forward is generally easier than moving two, which is easier than moving three. Waiting until something reaches end-of-life means you're multiple versions behind, increasing the likelihood that functionality you rely on has been deprecated or removed.

The same principle applies across development. Finding problems in development is cheaper than finding them in QA, which is cheaper than finding them in production. Security follows this curve most dramatically. Addressing vulnerabilities during development is vastly cheaper than responding to a public security breach.

When discussing technical debt with business stakeholders, this compounding cost curve is the most persuasive argument for proactive management.

Making Technical Debt Visible¶

Creating organizational awareness around technical debt is essential for managing it effectively. The approaches I've found most effective include visualization tools that make the abstract concrete.

One simple but powerful approach that worked for us was a "stoplight" system for technical debt. We identified all libraries, frameworks, and language versions in use and tracked their end-of-life dates. These became our "red light" dates - points at which we must update or risk being on unsupported versions. This system helped build a clear priority list and highlighted whether manual maintenance was feasible or automation was needed. Managing 5 dependencies manually is reasonable; managing 500 is not.

Automation as Visibility¶

Tools like Dependabot provide ongoing visibility into dependency status. When automated systems continuously generate pull requests for outdated dependencies, the entire organization develops awareness of technical health. These systems make the invisible nature of tech debt visible and turn theoretical discussions about debt into actionable items. It also does it gradually, so it just becomes part of system maintenance, instead of giant maintenance cycles that slow development.

When discussing technical debt with non-technical executives, I focus on two key aspects. First is why we need to address it. These reasons vary depending on the scenario and team, but could be something like us being on an unsupported version, or we that spend significant hours weekly working around issues fixed in newer versions.

The second aspect is considering the consequences of deferring. If we remain on an unsupported version, the vendor won't help when problems arise or at least not cheaply. There might be active exploits in the wild for the version of a library we are using.

I also include impact analysis on existing priorities. If properly planned, technical debt work fits into existing timelines with minimal disruption. When that's not possible, I clearly explain the trade-offs and impacts of both addressing and deferring the debt.

Cultural Elements of Technical Debt Management¶

Technical debt management isn't only a technical practice. It's a cultural one too. The most successful approaches I've seen embed debt management into everyday work rather than treating it as an occasional "debt sprint."

A culture of automating away low-level technical debt proved crucial in my experience. Implementing tools like Dependabot, OWASP scans, and accessibility checks reduced the time engineers spent worrying about these issues because they were handled behind the scenes. This automation-first approach served multiple purposes. It baked prevention into the development process, encouraged engineers to think about security, accessibility, and dependencies early, demonstrated organizational commitment to technical excellence, and created a positive ripple effect as other teams adopted similar practices.

When one team successfully automated technical debt management, others naturally followed suit. This created a culture of continuous improvement rather than periodic cleanup.

One particularly effective practice I found was assigning new engineers small technical debt tasks as their first projects. This might be a library update, minor bug fix, or addressing a potential security flaw. New engineers made immediate, valuable contributions while building confidence in a new system by completing manageable tasks. The team addressed technical debt items that might otherwise struggle for prioritization, documentation got updated based on fresh perspectives, and new team members learned team processes in a low-pressure context.

By introducing technical debt management during onboarding, we established it as a normal part of engineering work instead of an occasional activity.

The Continuous Approach¶

Rather than scheduling occasional "technical debt sprints" that disrupt normal work, I've found greater success in making debt reduction a continuous process.

Traditional technical debt sprints create several problems. They interrupt feature development momentum, treat technical excellence as exceptional rather than normal, and encourage deferring small fixes that could be handled immediately to "some other time".

Instead, I've found treating technical debt like regular household maintenance works better. By making small, continuous improvements, you prevent the need for major renovations in most cases.

With automated tools handling routine updates, engineering teams can focus on substantive technical debt such as infrastructure improvements, major framework upgrades, outdated UI refreshes, and performance bottlenecks. These larger initiatives deserve careful planning and dedicated time, but they should still be approached as part of normal development cycles rather than exceptional activities.

The most efficient technical debt management strategy I've witnessed is preventing unnecessary debt in the first place. During high-growth periods, implementing automated checks early in the development process caught potential issues before they became embedded in our systems. Clear standards for code quality, security, and architecture provided guidance without imposing unnecessary constraints. The key was establishing these standards as enablers rather than barriers. They helped developers make good decisions quickly rather than slowing them down with bureaucracy. It also eliminates back and forth during code review - "This line is 120 characters", "This function call should have parameters on new lines". Bake that into your linter that every developer must use and those discussions will never need to occur again.

Some technical investments I've seen pay dividends by preventing debt accumulation. Comprehensive test suites caught regressions early. CI/CD pipelines enforced quality standards. Architecture decision records preserved context. Service boundaries limited the spread of technical compromises. These investments created an environment where quality was the path of least resistance rather than an additional burden.

Conclusion¶

In my experience in multiple leadership roles, when managed effectively, technical debt becomes not just a necessary evil but a strategic tool. By making conscious decisions about when to take on debt and when to pay it down, companies can maintain the velocity needed for market success while building sustainable technical foundations.

I've found that making debt visible through automation, metrics, and clear communication ensures technical debt is part of strategic conversations. Embedding debt management in culture builds technical excellence into everyday practices rather than occasional initiatives. Focusing on prevention by investing in tools and practices makes it easier to do things right than to create new debt. Being strategic about remediation means addressing debt that limits key business initiatives while deferring debt that doesn't impact current priorities. Measuring the impact by tracking before-and-after metrics demonstrates the value of technical debt reduction.

As engineering leaders, our role isn't to eliminate all technical debt. That would be neither possible nor desirable. Instead, our job is to manage technical debt as thoughtfully as we would manage financial debt. The goal is to take it on deliberately when the terms make sense, monitoring its impact, and paying it down strategically to maximize long-term value.

Scaling Engineering Teams: From 5 to 50+ Without Losing Your Culture

2025-04-01T10:00:00-05:00

After nearly two decades in software development leadership and multiple VP-level roles, I've learned that scaling engineering teams is about much more than just adding headcount. It's about creating sustainable growth that preserves the core elements of your culture while evolving systems to support a larger organization.

The Challenges of Rapid Growth¶

When scaling engineering teams, I faced two consistent challenges that required thoughtful solutions:

Challenge 1: Accelerated Hiring While Maintaining Quality¶

Hiring quickly while finding candidates with the right skillsets is one of the most difficult aspects of scaling. The pressure to fill seats can easily lead to compromising on talent quality or cultural fit. I've found that being crystal clear about technical requirements while remaining flexible about where a candidate's talents can best serve the organization helps thread this needle.

During interviews, I focus on diving deep into candidates' past projects rather than relying solely on technical assessments. If an engineer discusses a project and emphasizes backend architecture and database design while only briefly touching on UI elements, this reveals their true areas of expertise. I always follow up by asking whether they want to continue developing in those areas or expand into new ones. Both answers are valuable—they help me understand not just what the candidate can do today, but how they might grow tomorrow.

Soft skills are incredibly important. An engineer that takes ownership of the system, is willing to try an unknown, likes mentoring (and being mentored), and can communicate with others in a professional and empathic manner can bring much more than their technical skills to a growing team. Obviously you need that technical skill, but an engineer isn't a machine. They should be able to do more than sling code, rebuild a server, or make a system diagram. I am hiring someone for their whole set of skills.

Challenge 2: Knowledge Transfer at Scale¶

In small teams, knowledge naturally diffuses through daily interactions. As teams grow, this organic process breaks down, creating knowledge silos and redundant work. Each wave of engineers we onboarded faced steeper learning curves without proper documentation and orientation. Our solution was to make each cohort of new hires part of the knowledge-building process. Every wave of engineers contributed to our knowledge bases, improving the experience for the next group. We implemented, essentially, an onboarding buddy system that paired new engineers with each other to build relationships, while also connecting them with experienced team members who could fill in knowledge gaps.

The other benefit of these types of improvements, is that it supports asynchronous communication and remote work. Remote work and hybrid work have grown in popularity, especially since 2020. Building knowledge transfers through tooling that doesn't expect the entire team to be in a single location provides so much context to others. Someone can catch up in a Slack channel later, or read a Confluence article tomorrow.

Culture Preservation Techniques That Work¶

Maintaining culture during growth requires intentionality. Here are strategies I've found effective:

Immediate Integration into Meaningful Work: New team members were immediately included in team ceremonies and assigned meaningful tasks—not just busywork. This sent a clear message: you're a valued contributor from day one. By giving new engineers real problems to solve immediately, they quickly identified with the team's mission and felt ownership of their work.
Public Knowledge Sharing: We instituted a rule that all support work must occur in public channels. This transparency accomplished several things. New engineers could observe troubleshooting in real-time allowing everyone to learn from each issue that arose. Team members could spot when someone was solving a previously addressed problem which helped find patterns of recurring issues. These became visible, highlighting systemic problems that needed permanent fixes.
Visual Documentation: A picture truly is worth a thousand words on day one. System architecture diagrams, workflow charts, and even team organization maps help new engineers orient themselves quickly. We continually updated these visual aids as our systems and teams evolved.

Evolution of Team Structure and Communication¶

Adaptive Team Structures¶

There's no one-size-fits-all approach to team structure during scaling phases. I've built dedicated backend, frontend, and QA teams, and I've also created cross-functional pods. The right approach depends on your product, business needs, and the existing strengths of your team. What matters most is acknowledging that your structure must evolve as you grow. The flat organization that worked at 5 engineers simply won't function at 50. Being transparent about these necessary changes helps the team understand why restructuring happens.

Communication Cascade¶

As teams grow, communication needs to flow through multiple channels. While I always strive to stay hands-on and close to the work, the reality of executive leadership means I can't directly communicate with everyone daily. Building a strong communication cascade through direct managers becomes essential.

I maintain regular one-on-one meetings with my direct reports and hold skip-level meetings on a consistent cadence. I also encourage my direct reports to do the same with their teams. This creates multiple pathways for information to flow up and down the organization. Crucially, engineers need to see their feedback traveling up the chain and resulting in actual changes. When team members witness their input driving decisions, they remain engaged in the communication process and feel valued for their insights.

Measuring Success Beyond Headcount¶

Adding engineers is easy—ensuring they contribute effectively is the real challenge. Here are metrics I've found valuable for measuring successful scaling:

Time to first meaningful commit: How quickly can a new engineer contribute value?
Feature deployment velocity: Is delivery speed maintaining or improving as the team grows?
New product launch timelines: Are we bringing innovations to market faster?
Support queue metrics: Are new engineers getting involved in support while maintaining quality and response times?

These indicators help identify whether your growing team is becoming more capable or just more numerous.

The Lesson I'd Apply to Future Scaling Efforts¶

If I were scaling a team again tomorrow, I'd place even greater emphasis on breaking down business expectations into clearly defined incremental goals. Saying "we're adding X engineers to deliver Y major initiative" creates unrealistic timelines and pressure. New engineers need time to contribute meaningfully to large tasks. By defining a granular priority list with visible intermediate steps, both new and existing engineers can march toward tangible milestones rather than an intimidating "big thing" with unclear pathways to completion.

Cross-Team Integration from Day One¶

Finally, I've learned the critical importance of ensuring engineering teams aren't isolated from other departments. As engineers onboard and learn, they should continuously interact with product, sales, customer success, and other functions. This cross-functional exposure provides vital context about why certain features are requested or designed in specific ways. Engineers who understand the business rationale make better technical decisions and can anticipate future needs. For example, understanding upcoming business requirements helps engineers build flexibility into systems where it will actually be needed, rather than adding unnecessary complexity. Building with the right level of flexibility now is infinitely easier than redesigning systems later.

Conclusion¶

Scaling engineering teams successfully requires balancing technical excellence, cultural preservation, and business alignment. By implementing thoughtful onboarding processes, maintaining transparent communication, measuring meaningful metrics, and keeping engineers connected to the broader business context, you can grow your team without losing what made it special in the first place. The journey from 5 to 50+ engineers is challenging, but with intentional leadership, it can result in not just a larger team, but a more capable, cohesive and impactful organization.

My first experiences with a laser cutter

2025-01-02T10:15:00-06:00

Introduction¶

My local library spent a lot of time and money remodeling the last few years. Part of this remodel was adding an Innovation Lab with different tools that the public can utilize. These include a Cricut cutter, a Glowforge, a couple 3D printers, sewing machines, a T-shirt screen printer, and a few others I can't recall off hand. I am excited to try out several of these, but started with the laser cutter, because I have a home project that needs a few small things cut and engraved that I wasn't sure how I was going to do.

The end goal is to get a set of two inch by one inch rectangles with various years engraved, scored or cut so that I can use these as labels.

A simple start¶

After a bit of research, I found that what I wanted was relatively simple - at least compared to what the machine is able to do. To do more advanced things, I'd have to learn a lot more about how some software tools operate. For my project though, I used Inkscape. I started by setting up the rectangle I wanted to use, rounding the corners, and adding a year to engrave. This became my basic template, to ensure that everything was the same size.

In all of my SVG files, I only used three colors:

Red for cuts
Black for engrave
Blue for scoring

While I read the Glowforge can handle different colors, I noticed that other tooling and software could not and used these three colors. If I ever get access to another laser cutter - or buy one of my own - I don't want to have to redo all of my work.

The important thing at this step was to select the text, then Text->Object to path. Without this step, the Glowforge application didn't see the text, just the rectangle.

Getting Fancy¶

Once I had a basic cut done on basswood proof grade material from Glowforge, I wanted to try a few other designs. All of my examples below are using 2009, because that's the set I grabbed when taking pictures.

With this, I engraved the background, instead of the text, so that the year would pop out. I didn't end up liking this one very much, because it makes the material so much thinner. This is obvious when I had multiple tiles of years that had not all been engraved this way. The thinness made these specific tiles feel flimsy. Since I didn't want to use this pattern for all years for the project, it wasn't going to work for only a couple.

I tried a couple acrylic colors - Black and Blue - to see how it'd look. In this test, I wanted to deal with the thinness I mentioned above too. I did this by adding a small border around the outside, so that when the tiles were side by side, the outer edges were all the same thickness. I liked how this looked and would end up using a portion of this in some final designs. The black acrylic turned out well.

Unfortunately, I didn't like the blue acrylic. This is how blue looks with the masking tape still in place. This tape is to prevent scorching marks. It looks ok with the tape in place, but once removed, the blue text just kind of got lost.

This test didn't have the outer border, but I don't think that would have helped me like it any better. The blue text just gets lost from any angle other than straight on.

I cut out individual numbers as well. These look really good. Unfortunately, I didn't plan beyond the cut out, and dealing with four individual numerals on each item I wanted to label very quickly became a ton of work to ensure they were unmasked, aligned, and glued into place. They also didn't end up looking as good as the tiled items once I had them in place.

Something I should have learned from this experiment that I was vaguely aware of, but didn't pay attention to because I was focused on the numbers themselves, was that the inner part of the numbers would come out. This was more important in the next two experiments.

At this point I had decided that I liked how the black acrylic looked, but wanted to highlight a few years. I attempted one last test to see if I could utilize the narrower engraved look and stack it on top of another. The original goal had been to end up with the same thickness as the other tiles, but that didn't work as planned.

That said, for the few years that I needed to highlight, this worked very well. To build this, I kept the lower layer - the wood - the same size as other tiles. I engraved like I had done for the acrylic tests above. Then on the acrylic itself, I made it the size of the inner border and reversed the numerals. I did this because I originally tested by engraving the acrylic down to size too, but it didn't look good. Instead, I kept the acrylic the original width and just cut out the numbers.

Once stacked and glued together, I realized that I hadn't kept the inner pieces of the zeros. It is easy enough to go cut out a couple more numbers at the library, but at the same time, not bothering me enough to do so.

Conclusion¶

Over the holidays, I completed the project and got everything labeled. I'm pleased with how it turned out. I was not surprised at how easy the Glowforge itself is to use. Learning how to design the SVG cut files took longer than I expected. There are likely other software tools that could do the job better, but for this small project and for some experiments, Inkscape worked just fine.

For my next project, I'd like to finish off the WLED work I did last summer. The original project didn't work as expected, but the WLED portion worked wonderfully. I could build a couple stand lights and laser cut out the base. I'd have to figure out how to model the aluminium rail to cut it correctly, but that sounds like a fun task.

Moving MS Authenticator to a new phone without a personal account

2024-11-27T12:15:00-06:00

Introduction¶

Work utilizes the Microsoft Authenticator app as their 2FA application of choice. It's not my favorite, but it does what it needs to do, and since I only use the application for work things, it's nice to keep work accounts out of personal accounts in my other 2FA application. Recently, I got a new phone, and needed to migrate everything over to that device. As an aside, Android continues to make this easier and easier.

Standard advice on how to migrate your token to a new device is to enable cloud backup, sign in to the new device and import the backup. The problem is when you go to enable Cloud Backup from the settings, you are hit with this message:

"You need a personal Microsoft account to use cloud backup"

Well, I don't have a personal Microsoft account.

Aside: I can't provide screenshots of the Android application for this post because the application prevents screenshots from being taken. I appreciate this feature as part of the application for security purposes, but it's annoying when trying to write about it. Oh well...

Alternative to Cloud Backup¶

I have set up MFA on my work account - obviously - and can access settings through the Microsoft MFA setup page. From here, you'll be on the Security Info page. It will show the current authenticator device being utilized. You'll need the old device during this process to authenticate one last time.

Click on "Add sign-in method" and then "Microsoft Authenticator"

On your phone, ensure you have the MS Authenticator application installed. Then follow the on screen prompts between your computer and your new phone. You'll select "Work or School" and then scan the QR code that is common with 2FA/MFA applications.

The last step in the process will be to send an authentication request to the new device. Do so, enter the verification code, and your new device will be added.

I recommend you wrap this up by removing your old device from the Security Info section now that the new device is approved.

Conclusion¶

The Microsoft authenticator application doesn't make moving to a new device as easy as some of its competitors. The requirement for a personal Microsoft account to go through the most commonly recommended way is also a non-starter if you don't have such an account already. Fortunately, the alternative posted above ended up working just fine. I assume I'll need a new phone again, and this should help at least me, remember that there is an easy enough way to migrate.

Format USB drive with EFI partition in Windows

2024-11-13T13:00:00-06:00

Introduction¶

I seem to collect USB drives. They are handed out at conferences. They are used as recovery media for new machines. They just randomly appear on my desk. I use them to build live Linux CDs, transferring media from one device to another, building recovery media, and generally the things you'd expect a USB drive to be used for. Generally, when I use a USB drive the first thing I do is format it. My USB drives are not for long term storage and I consider anything on them used one time then the drive goes back to the bin for reuse another time.

An annoyance with this, though, is that when I use a drive to build a bootable backup media, often times an EFI partition will be created. As soon as this is created, Windows prevents that partition from being deleted in the future without taking additional steps. I have to go look those up every time because I do it infrequently enough to remember. This is a simple walkthrough of my process to solve this problem

A USB drive with an EFI partition looks like this within the Disk Management tool in Windows

Format an EFI Partition¶

To remove this partition, utilize the built in diskpart tool. Win + R and type in diskpart and allow the application to run.

First you need to find which drive you want to adjust partitions on.

Attention

Ensure you select the correct disk. Failure to do so could result in an inoperable system

Plug the USB drive into the machine and run list disk to see all available disks.

DISKPART> list disk

Disk ###  Status         Size     Free     Dyn  Gpt
--------  -------------  -------  -------  ---  ---
Disk 0    Online         3726 GB  1024 KB        *
Disk 1    Online          931 GB  1024 KB        *
Disk 2    Online           14 GB    14 GB

You can see that I have 3 drives detected. Disk 2 is my USB drive.

Select the disk.

DISKPART> sel disk 2

Disk 2 is now the selected disk.

Then show the partitions on the disk.

DISKPART> list partition

Partition ###  Type              Size     Offset
-------------  ----------------  -------  -------
Partition 1    System            4000 KB   850 KB

There is a single partition - the EFI partition. The rest of the drive is unallocated, as the screenshot above shows. Select this partition. If you have multiple partitions listed, you'll need to determine which is the EFI partition. In my case, since I'm going to be formatting the entire drive anyway, if there were multiple listed I'd select and delete each individual partition.

DISKPART> sel partition 1

Partition 1 is now the selected partition.

Attention

These commands will delete a partition. If you haven't selected the right drive or partition things will break.

Finally, delete the EFI partition

DISKPART> delete partition override
DiskPart successfully deleted the selected partition.

From the disk management tool, you can verify it's gone and format as normal.

Controlling power to LED lights with WLED Controller and a Relay switch

2024-08-16T12:00:00-05:00

Introduction¶

It's time to continue my summer project of setting up some outdoor LED lights. Previously, I set up and updated the Ericsity controller with WLED.

I am putting up approximately 20 meters - 65 feet - of LEDs. Total this will be nearly 2,000 individual LEDs and approximately 665 individually controlled LED segments (3 LEDs per segment). This will take more power than the little controller can handle. To demonstrate the problem of powering all of these LEDs with only the controller, look at this image:

These strips are wired with the end of one strip connected to the start of the next. The top strip is connected directly to the controller. The one below is the end of strip two, followed by the start of strip three and the bottom is the very end of the full run of LEDs. The lights are all set to the same color, but as you can see they clearly aren't the same color. The voltage drop across 65 feet of LEDs means that the LEDs are the end can't get enough power to match their earlier siblings.

Power Injection¶

The solution to this specific problem is to inject power into the LED strips. I picked up some WAGO connectors and a NUOFUWEI power supply. I also had some 18 gauge wire on hand. With this power unit, I can easily set up three injection points.

The goal was to inject power at the start of strip 1, right where the controller is connecting in the image above. Then inject between strips 2 and 3, in the middle of the run. Finally, I injected power at the end of strip 4. With these three, equally spaced injection points, I was able to get a nice uniform color across the entire 20 meter run.

Problem Solved! Right?

Not exactly.

Turning off the PSU¶

WLED controls the lights, but doesn't control the power supply with this wiring setup. Turning off the lights within WLED does turn off the LEDs, but the power supply continues to run. It's not drawing at full load, but it is drawing power and the cooling fan is active. It's noticeable and unneeded. I want a way to turn off the power supply AND the LEDs at the same time.

I can't power the WLED controller from the large power supply to do this, because if I turn off the power supply that'd also turn off the WLED controller. I'll need to power the WLED controller independently from the lights. Fortunately, this won't be a problem.

The next step is figuring out how I can use WLED to control the larger power supply. Fortunately, WLED has the ability to control a relay, which I can use to control the power supply. The Ericsity controller also has two output data pins. While I don't think the second one was built in to control a relay, it works perfectly here.

Wiring¶

I purchased a HiLetGo relay so that I could toggle the larger power supply on and off. To do this, it's important that the data line and the ground are common among the controller, the power supply and the LEDs.

The diagram above is a rough schematic of how I wired this. The controller sends data, but not power, to the LEDs. Data was on GPIO 16. The important part here is that the data line is shared across all of the strips. I did not need a signal booster for my project, and because I'm about to use the second exposed data channel for the relay, I had to ensure that this single channel could send a signal down the entire length of the strip. Fortunately, I didn't have any issues.

The wiring ground was tied into the PSU and the LEDs as well. The common ground is important.

On the other side of the diagram is the relay. I put this relay between the wall and the PSU. When WLED sent an ON signal, it would close the relay, turning on the PSU and the LEDs. GPIO 2 was tied to the relay and within LED Preferences, the Relay Pin was set to GPIO 2.

One quick power cycle from within WLED is required at this point. Press the power button in the UI to turn everything off, press it once more to turn it on. As long as the PSU is plugged into the wall, you should hear it fire up and see the LEDs turn on. If you press the power button again, the LEDs turn off, the relay clicks, shuts down the PSU and only the WLED controller remains active.

Success¶

With the relay in place, the LEDs don't draw phantom power while off because the PSU isn't active. The added benefit, at least for this specific power unit, is that the fans aren't running constantly so it's not as loud. While this will eventually be outside, I'd still prefer to not hear the fan when the lights are off. While they are active it's not going to be bothering me, because I'll likely have music playing for the sound reactive features which will easily be louder than this fan.

The next step in the project is going to be to get this set up outside.

Setting up and updating the Ericsity WLED Controller from 0.13.3 to 0.14.1

2024-06-20T15:45:00-05:00

Introduction¶

My summer time project this year is to set up some LED lights outside. After testing out a couple Govee strips that I didn't like because I couldn't diffuse the individual lights away, I finally found an LED strip I liked. The next step was to control these strips so that I could do more than solid colors or the default rainbow every LED strip has. I settled on using the amazing WLED project to control the lights. To speed up the project, I decided to get a prebuilt controller and eventually selected the Ericsity controller with a built in mic.

Side note: This post is just about how I updated the controller. The next article in the post is available, and talks about how I am controlling a larger power supply with the WLED controller.

The Problem¶

The controller arrived and was easy to set up. I wired up my strips in the basement to ensure I didn't have any power problems and then determined where I needed to make cuts and solder points once this was outside. Then I went to set up segments in WLED so that I could control effects in specific areas.

The problem I quickly encountered was that I am outlining my deck with these lights and right in the middle of the deck is a set of steps that I also wanted to light up and then the rest of the deck rail. WLED doesn't have the ability (at least that I've found) to span an effect across multiple segments and I'd need multiple segments for this.

At minimum I'd need 3 segments - The rail, the steps, the rest of the rail. If I want an effect to span the entire rail, I can't do that with the steps in the middle. Further research pointed me to the ability to remap the physical LED order to a logical LED order. After trying to set this up, it wasn't feature complete in 0.13, but did appear to be in 0.14.

However, the Ericsity - rightfully so - utilizes a stable version of WLED that only allows you to update to 0.13.4 and the README has the following note:

This repository is still maintained, and will receive bugfixes. However no new features will be added.

Setting up Ericsity Controller¶

Let's take a step back and get the controller set up and then focus on upgrading it. Fortunately, Ericsity makes this pretty easy.

Step 1 is to plug in the controller. After a few seconds, you will see a new network available on your phone called WLED-AP. Join that network. From your browser navigate to 4.3.2.1 and select "WIFI SETTINGS".

The goal is to add this controller to your wireless network. Do this by adding your network name and password into the first two text boxes. I also updated the mDNS setting, but that's optional. Then save and connect.

Finally, install the WLED mobile application if you are going to manage this via your phone. On an android device you can find it in the Google Play store with the name WLED. Reconnect to your wireless network and the one you just added the controller to. Once installed and reconnected to the network, open the application.

If you have already installed and configured a controller, you'll see your previous controllers listed. To add the new controller, click on the + sign in the upper right. Then "Start Discovery". When the controller is found, you can press the check button.

The new controller will be listed on the main WLED page with the default name of WLED-SoundReactive. Select that controller by tapping it. Then select "Config" along the top and "Security & Updates" at the bottom.

Scroll all the way down to the bottom of this page and you should see that you have the following version installed:

WLED SR version 0.13.3

Update to 0.14.x¶

From the Security & Updates page, it makes sense to try and perform an update. If you click "Manual OTA Update" you'll notice that only version 0.13.3 is available via this link though. It's time to do a manual update!

I am utilizing the MoonModules branch because the README for the default install says that changes should be made against this branch.

Pull Requests should be created against the MoonModules mdev branch.

I performed this OTA upgrade from my computer by navigating to the mDNS I set up previously. It can also be accessed by the IP address of the controller in your browser.

At the time of this post, the current release was 0.14.1-b30. This was released approximately 6 months ago. I briefly skimmed through recent issues and pull requests to see if I should find a more recent build. There was a crash issue reported with the ripple effects. Since I like that particular effect, I decided to go with a newer build than the official release.

Official build¶

If you are going to stick with an official release, navigate to the release page and download the binary file you want to install. For the Ericsity, I found that the generic ESP32 build worked. If you want to use this, look for the file titled WLEDMM_0.14.1-b30.36_esp32_4MB_M.bin. You could also use the WLEDMM_0.14.1-b30.36_esp32_4MB_S.bin. Download this file.

Selecting a more recent build¶

Since I wanted a fix for the item I found, I opted to download a recent build of the mdev branch. This can be accomplished by navigating to the build pipeline and filtering for the mdev branch. This is on Github and available via this direct link to the mdev pipeline.

From here, select the build you want and scroll all the way to the bottom of the page. Find the binary you want to download. In my case, I wanted the firmware-esp32_4MB_M and download it. Once downloaded, extract it so that you can upload the .bin file within the WLED page.

Press the button!¶

In WLED's Security and Update page, select this binary and then click "Update!"

The update took less than a minute for me. While the new image was being installed, I was told not to close or refresh the page.

When the update is complete, the controller will reboot.

To confirm that everything has been updated, click on "Config". Immediately, you'll notice a lot more options. Scroll to the bottom and select "Security & Updates" and at the bottom you'll have an About section that lists the version you selected to install. Mine looks like this:

WLEDMM version 0.14.1-b31.38 ☾

Fixing Sound Reactive¶

During testing, I noticed that the sound reaction wasn't working. I found the AudioReactive plugin was disabled by default. This is an easy fix. In the WLED application (or web page), click on "Info" then click the power button icon next to "AudioReactive". However, this isn't enough to solve the problem. Click on "Config" then scroll down to AudioReactive. First make sure it is enabled (it should be after the power button selection above).

The digitalmic section needs to be modified. The Ericsity controller has the microphone on the following pins:

Microphone Type: Generic I2S
Pin I2S SD: 26
Pin I2S WS: 5
Pin I2S SCK: 21

I found that pin 5 was selected for another plugin. If you find this too, navigate back to Config page and select "Rotary-Encoder". I don't utilize this plugin, but it is part of the M build I downloaded. Disable the plugin and change the CLK Pin to be undefined. Then save this change and go back to the AudioReactive plugin and set it up with the pin layout above.

Finally, reboot the controller by selecting "Info" and rebooting the controller at the bottom.

Good to go¶

Now that the controller has rebooted, audio reactions are working again. The branch I've updated my controller to is actively maintained for the 0.14 branch, even if it is several months behind the main line. I'm satisfied with this for now and will begin experimenting with the led mappings that I need that started this entire upgrade process.

Python Gotcha: strip, lstrip, rstrip can remove more than expected

2024-03-29T09:00:00-05:00

Introduction¶

As a software engineer, you've cleaned your fair share of dirty strings. Removing leading or trailing spaces is probably one of the most common things done to user input.

In Python, this is done with the .strip(), .lstrip() or .rstrip() functions and generally looks like this:

>>> "     Andrew Wegner     ".lower().strip()
'andrew wegner'
>>> "     Andrew Wegner     ".lower().lstrip()
'andrew wegner     '
>>> "     Andrew Wegner     ".lower().rstrip()
'     andrew wegner'

That's pretty straightforward and nothing unexpected is going on.

Gotcha¶

The Gotcha is that each of these functions take a list of characters that can be removed.

>>> "Andrew Wegner".lower().rstrip(" wegner")
'and'

What happened? Why wasn't the result just

'andrew'

Explanation¶

Read the line from the documentation again, carefully:

A list of characters

Not a list of strings.

This is explicitly spelled out in the documentation, with an example, showing what the implications are. However, for a new developer, it's unexpected behavior. After all, these seem like intuitive functions.

The example above does the following:

Receives a list of characters to remove. In this case it is all letters in my last name, plus the space character: wegner
Lower case all letters in the input string, resulting in andrew wegner
From the right hand side of the string, begin removing characters that are in the input list. Stop when you encounter a character not in the list. In this case that means that rengew wer are removed (right to left) and then the d in andrew is encountered so that rstrip function stops.
Return the remaining string of and

Solution¶

Python has two functions that will correctly remove a string - .removesuffix() and .removeprefix() for right and left side removals.

>>> "Andrew Wegner".lower().removesuffix(" wegner")
'andrew'

These two functions were introduced in Python 3.9 as part of PEP-616. In the PEP, it explicitly calls out the confusion users have about the *strip() functions and how they behave. These two were introduced to allow the desired behavior.

One important note is that these two remove* functions will only remove at most one instance of the string.

>>> "Andrew Wegner Wegner".lower().removesuffix(" wegner")
'andrew wegner'

Installing the Mobile Security Framework (MobSF) on Windows using Docker

2024-02-19T15:00:00-06:00

Introduction¶

I recently needed to poke around an APK. I'd done this sort of thing before, but it's been years so the process of setting up an Android environment was unfamiliar. Instead of spending a day reimmersing myself for what was (hopefully) going to be a 10-15 minute thing, I turned to another tool: Mobile Security Framework (also known as MobSF)

Setting up Docker¶

There are countless blog posts, articles, and YouTube videos on how to set up the perfect docker environment. This was a quick task. I didn't need "perfect", I needed good enough. At the time, I was on a Windows 11 machine, so that's what I had to work with. The goal was to get Docker operational as quickly as possible.

Why? MobSF has a quick setup with two lines of Docker code.

Using the Install Docker for Windows documentation, I downloaded the installable and started the process. The installation took a couple minutes on the device I was on and required me to log out and back in. Total install time was less than 5 minutes.

Installing MobSF¶

Once logged back in, I fired up the command line. The MobSF README provides two lines to execute on the command line to pull the latest image. I didn't go through the process of setting up a dynamic analyzer. I didn't think I'd need it (and as luck would have it in this case, I was correct).

Run the two commands:

docker pull opensecurity/mobile-security-framework-mobsf:latest docker run -it --rm -p 8000:8000 opensecurity/mobile-security-framework-mobsf:latest

This will pull the latest version of MobSF and then run it on port 8000 on the local machine. Once it's running you can access it in your browser by going to http://127.0.0.1:8000

Find your APK¶

Theoretically, you'll be running this against an APK you've built and developed. In that case, you'll have the APK handy and can load it into the UI. I, unfortunately, didn't have that luxury. Fortunately, getting the APK is relatively simple.

Go to the Google Playstore and search for the application
Navigate into the details page of the application (you should see the option to install it)
Copy the URL to this page
Go to APKCombo
Paste the URL you copied in step 3 into the appropriate text box and click "Generate Download Link"
After a few seconds, click the download icon next to the link that was generated

You now have the APK to use.

Using MobSF¶

With the APK in hand, and MobSF running, navigate to http://127.0.0.1:8000. Upload the APK and wait a couple of minutes. If you are watching the logs, you'll see it performing multiple steps behind the scenes. Once it's done analyzing the APK, it'll drop you to the dashboard.

The dashboard has a lot of detail about the APK. Scroll through to see the permissions it's utilizing, certificates attached to it, potential security vulnerabilities, code analysis, detected URLs in the code, hard coded secrets, and strings. All of these would help a team build a better and more secure mobile application.

Conclusion¶

For my purpose, this 15 minute exercise not only answered the question I was seeking it also raised a few items that will need to be corrected before the next release. Introducing security practices with a tool like MobSF would be incredibly beneficial to a team tasked with building an Android application. Further, adding in the dynamic analysis capabilities can provide more details on areas to improve.

While I didn't need those to answer my question, I'd encourage someone looking to introduce this to a team, to look at the capabilities of MobSF

Review of Data Analysis with Polars Udemy course

2024-01-29T11:30:00-06:00

Introduction¶

After the previous course about Polars, I was excited to learn more. I settled on Data Analysis with Polars with Liam Brannigan instructing. I was drawn to this because of the mention of visualization using Matplotlib, Seaborn, Altair and Plotly. The course is billed at 2.5 hours long and is currently going for $85, though I did pick this up during one of Udemy's many sales.

Let's get started on the review and why I rated it so low.

Course¶

I have multiple complaints about this course.

The entire course is reading from pre-built notebooks or slides.
The last 26 out of 27 lectures are one page slides.
The quizzes in the middle of the courses are not effective, because the course is not designed for the learner to actually do any coding.
The exercises at the end of lectures are never revisited.

This course felt like the instructor was given a bunch of Jupyter notebooks by a more senior instructor and told to go teach the class. There is a lot of reading directly from the notebook, quickly glossing over code, and then talking through the output of the code. Unlike the last course, this one doesn't even feel like a cookbook of usefulness. It's just a glorified slide deck.

Let me show you what I mean. This is the course list's last 4 sections. I've collapsed them to show only the section header so it's a reasonable sized screenshot.

There are 27 lectures in these 4 sections. The total estimated time for these 27 lectures - 7 minutes. There is only 1 video in this entire group of sections. The other 26 out of 27 lectures are one page slides. The slides are "What you'll learn by the end of this lecture" slides and a link to a Jupyter notebook.

The quizzes scattered through out the course are also ineffective. Many of the questions asked are asking what code provided is correct. These can be helpful, if the learner has done any hands on work during the course. But, this course isn't designed that way. It's an instructor reading slides, hand waving at some code, and then executing the code to show it works. It's ineffective.

The same problem holds true with the exercises at the end of many lectures. They are designed to get the student more experience and I started doing them early in the course. But, they are never revisited. The instructor doesn't talk about them. It's just homework that's assigned as busy work, essentially.

Conclusion¶

Avoid this course. Nearly half the course is just a single slide for the lectures. The items I was interested in learning about - visualizations - were among those one page slides. The course is designed like a glorified README, and its not worth the sales price I paid, let alone the full price.