Andrew Wegner | Ponderings of an Andy - technical

Review of Claude Code for Python Developers from Real Python

2026-04-06T10:30:00-05:00

Introduction¶

If you are in the software industry and haven't started working with an AI assistant by this point in 2026, then you should start getting concerned about your role in the company you work at. Throughout the first few months of 2026, there have been several large layoffs by major corporations. Companies like Block and Oracle cited AI as the driver.

AI is here and as of today, it's the least capable it will ever be, because each day it gets better. I've been exploring it since ChatGPT came out and it's impact on interviews. I've written about it multiple times recently. AI's impact on junior developers is particularly important because I'm already seeing this in my role.

My teams have been utilizing a handful of AI assistants over the past year and making amazing improvements to our workflows. From things as simple as reducing the SDLC cycle time for major features to triage of support items, the impact has been dramatic.

I feel like I've only touched the surface - not even scratched it - touched the surface of how to better use AI tooling, and I've been using it a lot. So, I looked for a workshop to increase my knowledge. As a subscriber to Real Python, I was happy to see they offered a brand new workshop about Claude Code and as of this writing they are offering it again.

The course - Claude Code for Python Developers: Hands-On Agentic Coding Course. A bit wordy, but an amazing two days.

Spoiler¶

As a spoiler, my review of the course is on the main course page.

"Learning about how Claude Code works was great. Working with things like Skills, learning a workflow that functions, was what I was hoping to learn about. All of those were covered."

"I feel more comfortable with the tool itself and how to implement a basic workflow for myself with ideas on how to extend it to a whole team."

"This is one of the best training sessions I've joined in the last year across multiple platforms."

— Andrew Wegner, VP Product at Zayo

Course¶

As you can see from above, I thought this course was great. It was a two day (over a weekend) live course with about 100 participants from around the world. Based on the chat during the course, the skill set of participants ranged from "new to Claude code but experienced developer" to "somewhat familiar with Claude Code and looking to do more with it". It was a nice mix of participants. The instructor, Philipp Acsany, did a good job of answering questions, sharing content, and ensuring everyone was able to follow along. This was a very hands on workshop.

Before the course began, set up instructions were sent out. I can not overstate how much this was appreciated, because that meant the course could assume that everyone has a functioning environment to work in. This saved so much time on basic questions and allowed the course to start with the interesting content, not a tutorial on how to install Claude Code, GitHub CLI, Python and uv, Git and Zoom.

Day 1 started with the very basics of Claude. Normally, I'd be annoyed by starting with such a basic concept, but Philipp kept it engaging and more importantly, showed some best practices using Claude to scaffold a new project that I hadn't seen. Thinking that this bodes well for the rest of the course, I eagerly followed along.

As Day 1 moved on, we built upon our scaffold to develop a small application, learning how to manipulate various aspects of claude, setting up prompts and skills to assist our workflow and learning how to debug when something doesn't work. Day 1 concluded with a little bit of homework. After 4 hours in the workshop, I felt pretty confident that I could accomplish this and was happy to see that confidence was justified. After about an hour more of individual work, I completed the tasks and was ready for day 2.

Day two built on top of the homework by adding in additional features, ensuring we were able to utilize various skills, and learning more about how Claude operates under the hood. The session concluded with quick demos of additional aspects of Claude - hooks, MCPs, and Agents. Honestly, this was my biggest disappointment in the course, because it showed so many things we wouldn't be getting to, but it did fill my "research later" queue.

Conclusion¶

This isn't a cheap course. But, it was worth it to me. I approached this with two goals in mind:

Learn something for myself that I could take and apply to personal projects
Learn something for my teams so that we could use it as inspiration to make further improvements to our workflows

Both of these were met in Day 1. That made Day 2 even more fun for me, because I showed up wanting to learn more, cover more, do more. As I said in my review on Real Python: This is one of the best training sessions I've joined in the last year across multiple platforms.

Through out the course, student questions were responded to - both live via Philipp and in chat via one of Philipp's partners or from other participants. I found this aspect of the course really valuable too. I did get a handful of questions answered during it, but I was able to provide answers as well.

So, why did I only give this a 9 out of 10? What's preventing that last star?

There are aspects of the course that were touched on so briefly that I think would have been useful to dive into. These are the topics that have ended up in my research queue - hooks, mcp services, agents, agent teams. I think these could have filled another 4 hour block, but this was a weekend course and didn't fit. I'll be watching for a session that covers these topics.

The other thing - the cost can be prohibitive to students. There is the cost of the course itself: $800 normally, $500 on sale when I joined. Plus the cost of Claude. It is recommended to get at least the Max plan which runs $100 per month. I agree with that. I think if I'd only gone with Pro, I'd have hit usage limits during the course.

That said, if you can afford it (or get work to cover it as training), this is worth it for both an introduction to Claude Code and to learn about features you likely aren't using to their full power. Even with this course, I don't think I am doing that yet, but I know what to research now to get better for personal usage and for team improvements.

Python Gotcha: Reusing Generators Returns Nothing

2025-07-22T09:00:00-05:00

Introduction¶

In the previous article, we looked at logging uncaught exceptions. Let's utilize the log output from that post for another common task: error log file processing. This example is going to be pretty simple, as the error log for the post is tiny, but in a production environment this could be hundreds of thousands of lines, or gigabytes in size. Potentially, that's a lot to shove into memory for processing. But that's where a generator can come in to help.

Generators do not store their results, instead they maintain state and yield the result back to the caller. This means each line in a log can be processed and returned, without loading the entire file.

Flashback¶

As a reminder from the previous article, the log file being used looks like this:

2025-07-14 22:30:44,061 __main__ INFO Application start
2025-07-14 22:30:44,061 __main__ CRITICAL uncaught exception, application will terminate.
Traceback (most recent call last):
File "/home/andy/main.py", line 31, in <module>
    main()
    ~~~~^^
File "/home/andy/main.py", line 27, in main
    logger.info(divide(a,b))
                ~~~~~~^^^^^
File "/home/andy/main.py", line 21, in divide
    return a/b
        ~^~
ZeroDivisionError: division by zero

There is one INFO entry and one CRITICAL entry.

yield vs return¶

It's important to understand what makes a generator different from a regular function. The key distinction is the yield keyword. When a function contains yield, Python treats it as a generator function, which behaves differently from functions that use return.

A regular function with return executes completely, returns back a single result and then terminates. A function with yield creates a generator object that can pause execution, return a value, and later resume from exactly where it left off. This is what enables the memory-efficient and lazy evaluation that makes generators powerful.

Gotcha¶

Log processing¶

A generator result can only be utilized one time. Whether you are using a generator to output the next item in a sequence or process a file line by line, once you have passed an iterable or exhausted the generator, it doesn't get reused. This simple generator demonstrates the issue.

def read_log_lines(filename):
    with open(filename, 'r') as f:
        for line in f:
            if 'CRITICAL' in line:
                yield line.strip()

error_logs = read_log_lines('app.log')

error_count = len(list(error_logs))
print(f"Found {error_count} CRITICAL lines")

recent_errors = [log for log in error_logs if '2025' in log]
print(f"Recent errors: {len(recent_errors)}")

At first glance, it looks like this will read the log file, count the number of errors and then output how many of those were in 2025 (or contain the string 2025). However, the actual output is different.

Found 1 CRITICAL lines
Recent errors: 0

error_logs is a generator object. If a function yields, it is a generator. As error_count is initialized, it processes the error log and yields back any critical lines. The list() function will consume the entire generator (file). A few lines later, the developer wants to see how many of these are recent errors and attempts to go through the error_logs generator again. Success! No recent errors!

Right?

No, and looking at the log quickly shows that.

Fibonacci¶

Let's use a generator to build the Fibonacci sequence. Spoiler for interviews! In this case, I'm going to use a generator to get the first 10 items. Then print out the first 5 and then try to print the entire list of 10 items.

def fibonacci(n):
    a, b = 0, 1
    for _ in range(n):
        yield a
        a, b = b, a + b

fib_numbers = fibonacci(10)

print("First 5 numbers:")
for i, num in enumerate(fib_numbers):
    print(num)
    if i >= 4:
        break

print("Entire List:")
full_list = list(fib_numbers)
print(full_list)

The output for this is:

First 5 numbers:
0
1
1
2
3
Entire List:
[5, 8, 13, 21, 34]

Notice that the full_list variable only contains the items remaining on the generator. Since the first 5 (indexes 0 through 4) were printed, they are no longer part of the generator. When the full list is printed, only the remaining items can be printed.

Solution¶

The solution to the problem is easy enough. Call the generator function again. For example, with the log code from above:

error_logs = read_log_lines('app.log')
error_count = len(list(error_logs))
...
error_logs = read_log_lines('app.log')  # Call again and create a new generator
recent_errors = [log for log in error_logs if '2025' in log]

For the Fibonacci code, you would call fib_numbers = fibonacci(10) again before printing the full list.

Obviously, there is a down side here with duplicate processing of the same data due to running the generator twice. This could probably be solved with some logic adjustments to the generator or the code calling the generator, but that'll vary by application depending on what the generator is doing.

Conclusion¶

The important thing to take away from this is that once you have iterated over an item in a generator, it's no longer part of a generator. This means that if you want to get clever and see if there are more items in a generator, or determine the next item, you've consumed the next item.

The power of generators, especially when processing large amounts of data, can't be understated. But, at the same time, it's important to know that reusing an exhausted generator or attempting to access a previous item directly from the generator is not going to work. Instead, to reuse generator logic, call the generator function again to create a new generator object, or convert to a list if memory permits and multiple iterations are needed.

Python Gotcha: Logging an uncaught exception

2025-07-14T23:00:00-05:00

Introduction¶

A well built application will use logging instead of print statements. An exceptionally well built one will log in such a way that additional context is added to each log message and be consumable by a log aggregation service. Perhaps I'll write up such an article in the future. For now though, let's focus on a single problem.

Here is some sample code to demonstrate the problem.

import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

handler = logging.FileHandler("app.log")
formatter = logging.Formatter("%(asctime)s %(name)s %(levelname)s %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)

def divide(a, b):
    return a/b

def main():
    logger.info("Application start")
    a = 10
    b = 0
    logger.info(divide(a,b))
    logger.info("Application end")

if __name__ == "__main__":
    main()

Briefly, this sets up the app.log file to receive our log messages. The main function is going to divide two numbers, log the result, and end the program. Pretty simple.

Except, in this case, it is dividing by zero. This throws an error and crashes the program.

The console spits out a stack trace

Traceback (most recent call last):
File "/home/andy/main.py", line 22, in <module>
    main()
    ~~~~^^
File "/home/andy/main.py", line 18, in main
    logger.info(divide(a,b))
                ~~~~~~^^^^^
File "/home/andy/main.py", line 12, in divide
    return a/b
        ~^~
ZeroDivisionError: division by zero

The app.log file contains a single line:

2025-07-14 22:20:04,551 __main__ INFO Application start

Gotcha¶

Where is the gotcha here? The stack trace is right there!

You are, of course, right. However, imagine that this was not a simple application, but instead a production application that sends logs to a central service. Your application crashed and no one was watching the console. Your app.log file has no information. It says the application started and then...nothing. What happened? Is it still running?

As you dig through running processes, or check a /health end point for responses, you find out that it isn't running. That took a lot of time, and production isn't responding.

You've lost all visibility to what happened in your application at the most critical moment. When it crashed and spit out a stack trace, you want as much detail as you can get.

Solution¶

The solution is sys.excepthook. This is called when any exception is raised and uncaught, except for SystemExit. It's pretty easy to utilize as well. A few small changes to the above code will allow us to log this completely unexpected ZeroDivisionError.

import logging
import sys

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

handler = logging.FileHandler("app.log")
formatter = logging.Formatter("%(asctime)s %(name)s %(levelname)s %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)

def handle_uncaught_exception(exc_type, exc_value, exc_traceback):
    logger.critical(
        "uncaught exception, application will terminate.",
        exc_info=(exc_type, exc_value, exc_traceback),
    )
sys.excepthook = handle_uncaught_exception


def divide(a, b):
    return a/b

def main():
    logger.info("Application start")
    a = 10
    b = 0
    logger.info(divide(a,b))
    logger.info("Application end")

if __name__ == "__main__":
    main()

The important bit is the new handle_uncaught_exception function and the sys.excepthook line (with appropriate import statement).

Someone running this in the console will notice that there is not a stack trace dumped to the console now. Instead, our log contains important information:

2025-07-14 22:30:44,061 __main__ INFO Application start
2025-07-14 22:30:44,061 __main__ CRITICAL uncaught exception, application will terminate.
Traceback (most recent call last):
File "/home/andy/main.py", line 31, in <module>
    main()
    ~~~~^^
File "/home/andy/main.py", line 27, in main
    logger.info(divide(a,b))
                ~~~~~~^^^^^
File "/home/andy/main.py", line 21, in divide
    return a/b
        ~^~
ZeroDivisionError: division by zero

This is great! Now when troubleshooting this failing application and looking at the logs, we can easily see that an exception occurred. Additionally, with proper updates to the logging, more context can be provided such as the values of a and b. While it's easy enough to figure out that b is 0 in this simple example, the context in a larger production application could save a ton of troubleshooting time.

Conclusion¶

Logging is vital to knowing what your application is doing. But, it's even more important to determining why it stopped working. If your logs aren't providing information when the application crashes, it's functionally useless. By implementing an excepthook, you can catch and properly log uncaught exceptions.

I know some of you are coming up with alternatives. Terrible ideas like wrapping the entire main block in a try/except. There are legitimate reasons to throw an exception. In this case, a ZeroDivisionError is a great exception to catch. But, you'd want to do it around as small of a code block as possible.

This is a clean way to catch truly unexpected exceptions, not something a developer could have anticipated and, perhaps, fixed with additional input validation.

How to fix GitLab 18 error regarding git_data_dirs

2025-07-06T08:00:00-05:00

Introduction¶

Several years ago I set up GitLab in my home environment. I appreciate past me documenting the basic steps I took to do this, because I'll need them again at some point when I make upgrades to my home lab. I've documented some things I've done (like setting up GitLab runners, or dealing with Grafana being deprecated within GitLab or utilizing GitLab to automatically backup my Obsidian notes) Unfortunately, I didn't document everything. One of those things that I didn't document was changing where my repositories are stored on disk by default.

In GitLab 17.8 (January 2025), a new deprecation warning started appearing.

git_data_dirs has been deprecated since 17.8 and will be removed in 18.0. See https://docs.gitlab.com/omnibus/settings/configuration.html#migrating-from-git_data_dirs for migration instructions.

I'm a little annoyed that there were only 5 months of notice on this because major versions are released in May. In any case, if you attempt to update to version 18 or beyond and are utilizing git_data_dirs, the upgrade will fail.

Solution¶

As usual, GitLab is good at providing documentation on resolving and migrating through the deprecations. However, on my first attempt I encountered an error. I believe it's because I missed a note buried in the text - not the code blocks - the first time.

Note that the /repositories suffix must be appended to the path because it was previously appended internally.

The solution is to open /etc/gitlab/gitlab.rb (you probably should make a backup first). Find the git_data_dirs line. For me this was around line 455. Then comment out this entire block.

Then find (or add) the gitaly['configuration'] block. This was immediately after the git_data_dirs section for me. Uncomment it, and add the appropriate path (from your git_data_dirs block) and add /repositories to the end of it.

Once you are done, run

gitlab-ctl reconfigure
gitlab-ctl restart

Give it a minute to fire up all the GitLab subcomponents, and then you should be able to update GitLab beyond version 18.

New code section¶

My gitlab.rb now has this:

#git_data_dirs({
#  "default" => {
#    "path" => "/previous/path/to/repos"
#   }
#})

### Gitaly settings
gitaly['configuration'] = {
storage: [
    {
    name: 'default',
    path: '/previous/path/to/repos/repositories',
    },
],
}

Conclusion¶

With this quick change, I can continue to hold the repositories at a location of my choosing. The biggest thing is to add /repositories to the path in the new Gitaly configuration. With this change, I can continue to utilize the current version of GitLab - a tool that I still find invaluable.

Python Gotcha: Identity vs Equality - When 'is' Fails Unexpectedly

2025-06-17T08:00:00-05:00

Introduction¶

In the comparisons gotcha I wrote a few years ago, I briefly touched on is vs ==.

Put simply, is should ONLY be used if you are checking if two references refer to the same object. Remember, is compares object references.

Even more simply, is is not checking value. Let's take a look at a couple examples.

Gotcha¶

Integers¶

Python, specifically CPython, caches the values of -5 through 256 (inclusive). This means that these small integer values will always refer to the same object.

Note the phrasing there - "the same object".

Outside of that range, though, the same is not true.

>>> a = 100
>>> b = 100
>>> a is b
True
>>> a = 257
>>> b = 257
>>> a is b
False

In both of the above examples, using a == b would have returned True. The mistake was assuming that is does the same thing. It does not.

Strings¶

String interning is a method of storing only one copy of each distinct immutable string value. Immutable strings can't be changed. Not every string will be interned though. Let's take a look:

>>> a = "Hello"
>>> b = "Hello"
>>> a is b
True
>>> a = "Hello World"
>>> b = "Hello World"
>>> a is b
False
>>> a = "Hello_World"
>>> b = "Hello_World"
>>> a is b
True

The first and last example interned the strings, showing that a and b refer to the same object. But, the second example - Hello World - didn't get interned, so a and b refer to different objects. Why is this?

The short and simply answer is that any string that has only numbers, letters or underscores will be interned. Since Hello World contains a space, it would not be interned.

The Solution¶

To a new developer that has seen tutorials that read if a is True or if b is None, a conditional for integers or strings following the same pattern appears to be comparing values. If they test it with small, positive numbers or simple one word strings, the assumption holds up.

But, == is for comparing values! Each of the above examples would return True by changing the statement to a == b.

The few times that is is appropriate are when you are checking True/False or None. Otherwise, the vast majority of the time, you want to use an equality check (==)

The Python PEP8 programming recommendations state:

Comparisons to singletons like None should always be done with is or is not, never the equality operators.

The linters in the Python ecosystem report on the usage of is vs == too. flake8 has E711 - Comparison to None should be 'cond is None:'. ruff has a similar report with it's None comparison check.

I highly recommend a linter for your projects to catch this, and other problems that go against best practices.

Remember, is compares object references, not object equality

My first experiences with a laser cutter

2025-01-02T10:15:00-06:00

Introduction¶

My local library spent a lot of time and money remodeling the last few years. Part of this remodel was adding an Innovation Lab with different tools that the public can utilize. These include a Cricut cutter, a Glowforge, a couple 3D printers, sewing machines, a T-shirt screen printer, and a few others I can't recall off hand. I am excited to try out several of these, but started with the laser cutter, because I have a home project that needs a few small things cut and engraved that I wasn't sure how I was going to do.

The end goal is to get a set of two inch by one inch rectangles with various years engraved, scored or cut so that I can use these as labels.

A simple start¶

After a bit of research, I found that what I wanted was relatively simple - at least compared to what the machine is able to do. To do more advanced things, I'd have to learn a lot more about how some software tools operate. For my project though, I used Inkscape. I started by setting up the rectangle I wanted to use, rounding the corners, and adding a year to engrave. This became my basic template, to ensure that everything was the same size.

In all of my SVG files, I only used three colors:

Red for cuts
Black for engrave
Blue for scoring

While I read the Glowforge can handle different colors, I noticed that other tooling and software could not and used these three colors. If I ever get access to another laser cutter - or buy one of my own - I don't want to have to redo all of my work.

The important thing at this step was to select the text, then Text->Object to path. Without this step, the Glowforge application didn't see the text, just the rectangle.

Getting Fancy¶

Once I had a basic cut done on basswood proof grade material from Glowforge, I wanted to try a few other designs. All of my examples below are using 2009, because that's the set I grabbed when taking pictures.

With this, I engraved the background, instead of the text, so that the year would pop out. I didn't end up liking this one very much, because it makes the material so much thinner. This is obvious when I had multiple tiles of years that had not all been engraved this way. The thinness made these specific tiles feel flimsy. Since I didn't want to use this pattern for all years for the project, it wasn't going to work for only a couple.

I tried a couple acrylic colors - Black and Blue - to see how it'd look. In this test, I wanted to deal with the thinness I mentioned above too. I did this by adding a small border around the outside, so that when the tiles were side by side, the outer edges were all the same thickness. I liked how this looked and would end up using a portion of this in some final designs. The black acrylic turned out well.

Unfortunately, I didn't like the blue acrylic. This is how blue looks with the masking tape still in place. This tape is to prevent scorching marks. It looks ok with the tape in place, but once removed, the blue text just kind of got lost.

This test didn't have the outer border, but I don't think that would have helped me like it any better. The blue text just gets lost from any angle other than straight on.

I cut out individual numbers as well. These look really good. Unfortunately, I didn't plan beyond the cut out, and dealing with four individual numerals on each item I wanted to label very quickly became a ton of work to ensure they were unmasked, aligned, and glued into place. They also didn't end up looking as good as the tiled items once I had them in place.

Something I should have learned from this experiment that I was vaguely aware of, but didn't pay attention to because I was focused on the numbers themselves, was that the inner part of the numbers would come out. This was more important in the next two experiments.

At this point I had decided that I liked how the black acrylic looked, but wanted to highlight a few years. I attempted one last test to see if I could utilize the narrower engraved look and stack it on top of another. The original goal had been to end up with the same thickness as the other tiles, but that didn't work as planned.

That said, for the few years that I needed to highlight, this worked very well. To build this, I kept the lower layer - the wood - the same size as other tiles. I engraved like I had done for the acrylic tests above. Then on the acrylic itself, I made it the size of the inner border and reversed the numerals. I did this because I originally tested by engraving the acrylic down to size too, but it didn't look good. Instead, I kept the acrylic the original width and just cut out the numbers.

Once stacked and glued together, I realized that I hadn't kept the inner pieces of the zeros. It is easy enough to go cut out a couple more numbers at the library, but at the same time, not bothering me enough to do so.

Conclusion¶

Over the holidays, I completed the project and got everything labeled. I'm pleased with how it turned out. I was not surprised at how easy the Glowforge itself is to use. Learning how to design the SVG cut files took longer than I expected. There are likely other software tools that could do the job better, but for this small project and for some experiments, Inkscape worked just fine.

For my next project, I'd like to finish off the WLED work I did last summer. The original project didn't work as expected, but the WLED portion worked wonderfully. I could build a couple stand lights and laser cut out the base. I'd have to figure out how to model the aluminium rail to cut it correctly, but that sounds like a fun task.

Moving MS Authenticator to a new phone without a personal account

2024-11-27T12:15:00-06:00

Introduction¶

Work utilizes the Microsoft Authenticator app as their 2FA application of choice. It's not my favorite, but it does what it needs to do, and since I only use the application for work things, it's nice to keep work accounts out of personal accounts in my other 2FA application. Recently, I got a new phone, and needed to migrate everything over to that device. As an aside, Android continues to make this easier and easier.

Standard advice on how to migrate your token to a new device is to enable cloud backup, sign in to the new device and import the backup. The problem is when you go to enable Cloud Backup from the settings, you are hit with this message:

"You need a personal Microsoft account to use cloud backup"

Well, I don't have a personal Microsoft account.

Aside: I can't provide screenshots of the Android application for this post because the application prevents screenshots from being taken. I appreciate this feature as part of the application for security purposes, but it's annoying when trying to write about it. Oh well...

Alternative to Cloud Backup¶

I have set up MFA on my work account - obviously - and can access settings through the Microsoft MFA setup page. From here, you'll be on the Security Info page. It will show the current authenticator device being utilized. You'll need the old device during this process to authenticate one last time.

Click on "Add sign-in method" and then "Microsoft Authenticator"

On your phone, ensure you have the MS Authenticator application installed. Then follow the on screen prompts between your computer and your new phone. You'll select "Work or School" and then scan the QR code that is common with 2FA/MFA applications.

The last step in the process will be to send an authentication request to the new device. Do so, enter the verification code, and your new device will be added.

I recommend you wrap this up by removing your old device from the Security Info section now that the new device is approved.

Conclusion¶

The Microsoft authenticator application doesn't make moving to a new device as easy as some of its competitors. The requirement for a personal Microsoft account to go through the most commonly recommended way is also a non-starter if you don't have such an account already. Fortunately, the alternative posted above ended up working just fine. I assume I'll need a new phone again, and this should help at least me, remember that there is an easy enough way to migrate.

Format USB drive with EFI partition in Windows

2024-11-13T13:00:00-06:00

Introduction¶

I seem to collect USB drives. They are handed out at conferences. They are used as recovery media for new machines. They just randomly appear on my desk. I use them to build live Linux CDs, transferring media from one device to another, building recovery media, and generally the things you'd expect a USB drive to be used for. Generally, when I use a USB drive the first thing I do is format it. My USB drives are not for long term storage and I consider anything on them used one time then the drive goes back to the bin for reuse another time.

An annoyance with this, though, is that when I use a drive to build a bootable backup media, often times an EFI partition will be created. As soon as this is created, Windows prevents that partition from being deleted in the future without taking additional steps. I have to go look those up every time because I do it infrequently enough to remember. This is a simple walkthrough of my process to solve this problem

A USB drive with an EFI partition looks like this within the Disk Management tool in Windows

Format an EFI Partition¶

To remove this partition, utilize the built in diskpart tool. Win + R and type in diskpart and allow the application to run.

First you need to find which drive you want to adjust partitions on.

Attention

Ensure you select the correct disk. Failure to do so could result in an inoperable system

Plug the USB drive into the machine and run list disk to see all available disks.

DISKPART> list disk

Disk ###  Status         Size     Free     Dyn  Gpt
--------  -------------  -------  -------  ---  ---
Disk 0    Online         3726 GB  1024 KB        *
Disk 1    Online          931 GB  1024 KB        *
Disk 2    Online           14 GB    14 GB

You can see that I have 3 drives detected. Disk 2 is my USB drive.

Select the disk.

DISKPART> sel disk 2

Disk 2 is now the selected disk.

Then show the partitions on the disk.

DISKPART> list partition

Partition ###  Type              Size     Offset
-------------  ----------------  -------  -------
Partition 1    System            4000 KB   850 KB

There is a single partition - the EFI partition. The rest of the drive is unallocated, as the screenshot above shows. Select this partition. If you have multiple partitions listed, you'll need to determine which is the EFI partition. In my case, since I'm going to be formatting the entire drive anyway, if there were multiple listed I'd select and delete each individual partition.

DISKPART> sel partition 1

Partition 1 is now the selected partition.

Attention

These commands will delete a partition. If you haven't selected the right drive or partition things will break.

Finally, delete the EFI partition

DISKPART> delete partition override
DiskPart successfully deleted the selected partition.

From the disk management tool, you can verify it's gone and format as normal.

Controlling power to LED lights with WLED Controller and a Relay switch

2024-08-16T12:00:00-05:00

Introduction¶

It's time to continue my summer project of setting up some outdoor LED lights. Previously, I set up and updated the Ericsity controller with WLED.

I am putting up approximately 20 meters - 65 feet - of LEDs. Total this will be nearly 2,000 individual LEDs and approximately 665 individually controlled LED segments (3 LEDs per segment). This will take more power than the little controller can handle. To demonstrate the problem of powering all of these LEDs with only the controller, look at this image:

These strips are wired with the end of one strip connected to the start of the next. The top strip is connected directly to the controller. The one below is the end of strip two, followed by the start of strip three and the bottom is the very end of the full run of LEDs. The lights are all set to the same color, but as you can see they clearly aren't the same color. The voltage drop across 65 feet of LEDs means that the LEDs are the end can't get enough power to match their earlier siblings.

Power Injection¶

The solution to this specific problem is to inject power into the LED strips. I picked up some WAGO connectors and a NUOFUWEI power supply. I also had some 18 gauge wire on hand. With this power unit, I can easily set up three injection points.

The goal was to inject power at the start of strip 1, right where the controller is connecting in the image above. Then inject between strips 2 and 3, in the middle of the run. Finally, I injected power at the end of strip 4. With these three, equally spaced injection points, I was able to get a nice uniform color across the entire 20 meter run.

Problem Solved! Right?

Not exactly.

Turning off the PSU¶

WLED controls the lights, but doesn't control the power supply with this wiring setup. Turning off the lights within WLED does turn off the LEDs, but the power supply continues to run. It's not drawing at full load, but it is drawing power and the cooling fan is active. It's noticeable and unneeded. I want a way to turn off the power supply AND the LEDs at the same time.

I can't power the WLED controller from the large power supply to do this, because if I turn off the power supply that'd also turn off the WLED controller. I'll need to power the WLED controller independently from the lights. Fortunately, this won't be a problem.

The next step is figuring out how I can use WLED to control the larger power supply. Fortunately, WLED has the ability to control a relay, which I can use to control the power supply. The Ericsity controller also has two output data pins. While I don't think the second one was built in to control a relay, it works perfectly here.

Wiring¶

I purchased a HiLetGo relay so that I could toggle the larger power supply on and off. To do this, it's important that the data line and the ground are common among the controller, the power supply and the LEDs.

The diagram above is a rough schematic of how I wired this. The controller sends data, but not power, to the LEDs. Data was on GPIO 16. The important part here is that the data line is shared across all of the strips. I did not need a signal booster for my project, and because I'm about to use the second exposed data channel for the relay, I had to ensure that this single channel could send a signal down the entire length of the strip. Fortunately, I didn't have any issues.

The wiring ground was tied into the PSU and the LEDs as well. The common ground is important.

On the other side of the diagram is the relay. I put this relay between the wall and the PSU. When WLED sent an ON signal, it would close the relay, turning on the PSU and the LEDs. GPIO 2 was tied to the relay and within LED Preferences, the Relay Pin was set to GPIO 2.

One quick power cycle from within WLED is required at this point. Press the power button in the UI to turn everything off, press it once more to turn it on. As long as the PSU is plugged into the wall, you should hear it fire up and see the LEDs turn on. If you press the power button again, the LEDs turn off, the relay clicks, shuts down the PSU and only the WLED controller remains active.

Success¶

With the relay in place, the LEDs don't draw phantom power while off because the PSU isn't active. The added benefit, at least for this specific power unit, is that the fans aren't running constantly so it's not as loud. While this will eventually be outside, I'd still prefer to not hear the fan when the lights are off. While they are active it's not going to be bothering me, because I'll likely have music playing for the sound reactive features which will easily be louder than this fan.

The next step in the project is going to be to get this set up outside.

Setting up and updating the Ericsity WLED Controller from 0.13.3 to 0.14.1

2024-06-20T15:45:00-05:00

Introduction¶

My summer time project this year is to set up some LED lights outside. After testing out a couple Govee strips that I didn't like because I couldn't diffuse the individual lights away, I finally found an LED strip I liked. The next step was to control these strips so that I could do more than solid colors or the default rainbow every LED strip has. I settled on using the amazing WLED project to control the lights. To speed up the project, I decided to get a prebuilt controller and eventually selected the Ericsity controller with a built in mic.

Side note: This post is just about how I updated the controller. The next article in the post is available, and talks about how I am controlling a larger power supply with the WLED controller.

The Problem¶

The controller arrived and was easy to set up. I wired up my strips in the basement to ensure I didn't have any power problems and then determined where I needed to make cuts and solder points once this was outside. Then I went to set up segments in WLED so that I could control effects in specific areas.

The problem I quickly encountered was that I am outlining my deck with these lights and right in the middle of the deck is a set of steps that I also wanted to light up and then the rest of the deck rail. WLED doesn't have the ability (at least that I've found) to span an effect across multiple segments and I'd need multiple segments for this.

At minimum I'd need 3 segments - The rail, the steps, the rest of the rail. If I want an effect to span the entire rail, I can't do that with the steps in the middle. Further research pointed me to the ability to remap the physical LED order to a logical LED order. After trying to set this up, it wasn't feature complete in 0.13, but did appear to be in 0.14.

However, the Ericsity - rightfully so - utilizes a stable version of WLED that only allows you to update to 0.13.4 and the README has the following note:

This repository is still maintained, and will receive bugfixes. However no new features will be added.

Setting up Ericsity Controller¶

Let's take a step back and get the controller set up and then focus on upgrading it. Fortunately, Ericsity makes this pretty easy.

Step 1 is to plug in the controller. After a few seconds, you will see a new network available on your phone called WLED-AP. Join that network. From your browser navigate to 4.3.2.1 and select "WIFI SETTINGS".

The goal is to add this controller to your wireless network. Do this by adding your network name and password into the first two text boxes. I also updated the mDNS setting, but that's optional. Then save and connect.

Finally, install the WLED mobile application if you are going to manage this via your phone. On an android device you can find it in the Google Play store with the name WLED. Reconnect to your wireless network and the one you just added the controller to. Once installed and reconnected to the network, open the application.

If you have already installed and configured a controller, you'll see your previous controllers listed. To add the new controller, click on the + sign in the upper right. Then "Start Discovery". When the controller is found, you can press the check button.

The new controller will be listed on the main WLED page with the default name of WLED-SoundReactive. Select that controller by tapping it. Then select "Config" along the top and "Security & Updates" at the bottom.

Scroll all the way down to the bottom of this page and you should see that you have the following version installed:

WLED SR version 0.13.3

Update to 0.14.x¶

From the Security & Updates page, it makes sense to try and perform an update. If you click "Manual OTA Update" you'll notice that only version 0.13.3 is available via this link though. It's time to do a manual update!

I am utilizing the MoonModules branch because the README for the default install says that changes should be made against this branch.

Pull Requests should be created against the MoonModules mdev branch.

I performed this OTA upgrade from my computer by navigating to the mDNS I set up previously. It can also be accessed by the IP address of the controller in your browser.

At the time of this post, the current release was 0.14.1-b30. This was released approximately 6 months ago. I briefly skimmed through recent issues and pull requests to see if I should find a more recent build. There was a crash issue reported with the ripple effects. Since I like that particular effect, I decided to go with a newer build than the official release.

Official build¶

If you are going to stick with an official release, navigate to the release page and download the binary file you want to install. For the Ericsity, I found that the generic ESP32 build worked. If you want to use this, look for the file titled WLEDMM_0.14.1-b30.36_esp32_4MB_M.bin. You could also use the WLEDMM_0.14.1-b30.36_esp32_4MB_S.bin. Download this file.

Selecting a more recent build¶

Since I wanted a fix for the item I found, I opted to download a recent build of the mdev branch. This can be accomplished by navigating to the build pipeline and filtering for the mdev branch. This is on Github and available via this direct link to the mdev pipeline.

From here, select the build you want and scroll all the way to the bottom of the page. Find the binary you want to download. In my case, I wanted the firmware-esp32_4MB_M and download it. Once downloaded, extract it so that you can upload the .bin file within the WLED page.

Press the button!¶

In WLED's Security and Update page, select this binary and then click "Update!"

The update took less than a minute for me. While the new image was being installed, I was told not to close or refresh the page.

When the update is complete, the controller will reboot.

To confirm that everything has been updated, click on "Config". Immediately, you'll notice a lot more options. Scroll to the bottom and select "Security & Updates" and at the bottom you'll have an About section that lists the version you selected to install. Mine looks like this:

WLEDMM version 0.14.1-b31.38 ☾

Fixing Sound Reactive¶

During testing, I noticed that the sound reaction wasn't working. I found the AudioReactive plugin was disabled by default. This is an easy fix. In the WLED application (or web page), click on "Info" then click the power button icon next to "AudioReactive". However, this isn't enough to solve the problem. Click on "Config" then scroll down to AudioReactive. First make sure it is enabled (it should be after the power button selection above).

The digitalmic section needs to be modified. The Ericsity controller has the microphone on the following pins:

Microphone Type: Generic I2S
Pin I2S SD: 26
Pin I2S WS: 5
Pin I2S SCK: 21

I found that pin 5 was selected for another plugin. If you find this too, navigate back to Config page and select "Rotary-Encoder". I don't utilize this plugin, but it is part of the M build I downloaded. Disable the plugin and change the CLK Pin to be undefined. Then save this change and go back to the AudioReactive plugin and set it up with the pin layout above.

Finally, reboot the controller by selecting "Info" and rebooting the controller at the bottom.

Good to go¶

Now that the controller has rebooted, audio reactions are working again. The branch I've updated my controller to is actively maintained for the 0.14 branch, even if it is several months behind the main line. I'm satisfied with this for now and will begin experimenting with the led mappings that I need that started this entire upgrade process.

Python Gotcha: strip, lstrip, rstrip can remove more than expected

2024-03-29T09:00:00-05:00

Introduction¶

As a software engineer, you've cleaned your fair share of dirty strings. Removing leading or trailing spaces is probably one of the most common things done to user input.

In Python, this is done with the .strip(), .lstrip() or .rstrip() functions and generally looks like this:

>>> "     Andrew Wegner     ".lower().strip()
'andrew wegner'
>>> "     Andrew Wegner     ".lower().lstrip()
'andrew wegner     '
>>> "     Andrew Wegner     ".lower().rstrip()
'     andrew wegner'

That's pretty straightforward and nothing unexpected is going on.

Gotcha¶

The Gotcha is that each of these functions take a list of characters that can be removed.

>>> "Andrew Wegner".lower().rstrip(" wegner")
'and'

What happened? Why wasn't the result just

'andrew'

Explanation¶

Read the line from the documentation again, carefully:

A list of characters

Not a list of strings.

This is explicitly spelled out in the documentation, with an example, showing what the implications are. However, for a new developer, it's unexpected behavior. After all, these seem like intuitive functions.

The example above does the following:

Receives a list of characters to remove. In this case it is all letters in my last name, plus the space character: wegner
Lower case all letters in the input string, resulting in andrew wegner
From the right hand side of the string, begin removing characters that are in the input list. Stop when you encounter a character not in the list. In this case that means that rengew wer are removed (right to left) and then the d in andrew is encountered so that rstrip function stops.
Return the remaining string of and

Solution¶

Python has two functions that will correctly remove a string - .removesuffix() and .removeprefix() for right and left side removals.

>>> "Andrew Wegner".lower().removesuffix(" wegner")
'andrew'

These two functions were introduced in Python 3.9 as part of PEP-616. In the PEP, it explicitly calls out the confusion users have about the *strip() functions and how they behave. These two were introduced to allow the desired behavior.

One important note is that these two remove* functions will only remove at most one instance of the string.

>>> "Andrew Wegner Wegner".lower().removesuffix(" wegner")
'andrew wegner'

Installing the Mobile Security Framework (MobSF) on Windows using Docker

2024-02-19T15:00:00-06:00

Introduction¶

I recently needed to poke around an APK. I'd done this sort of thing before, but it's been years so the process of setting up an Android environment was unfamiliar. Instead of spending a day reimmersing myself for what was (hopefully) going to be a 10-15 minute thing, I turned to another tool: Mobile Security Framework (also known as MobSF)

Setting up Docker¶

There are countless blog posts, articles, and YouTube videos on how to set up the perfect docker environment. This was a quick task. I didn't need "perfect", I needed good enough. At the time, I was on a Windows 11 machine, so that's what I had to work with. The goal was to get Docker operational as quickly as possible.

Why? MobSF has a quick setup with two lines of Docker code.

Using the Install Docker for Windows documentation, I downloaded the installable and started the process. The installation took a couple minutes on the device I was on and required me to log out and back in. Total install time was less than 5 minutes.

Installing MobSF¶

Once logged back in, I fired up the command line. The MobSF README provides two lines to execute on the command line to pull the latest image. I didn't go through the process of setting up a dynamic analyzer. I didn't think I'd need it (and as luck would have it in this case, I was correct).

Run the two commands:

docker pull opensecurity/mobile-security-framework-mobsf:latest docker run -it --rm -p 8000:8000 opensecurity/mobile-security-framework-mobsf:latest

This will pull the latest version of MobSF and then run it on port 8000 on the local machine. Once it's running you can access it in your browser by going to http://127.0.0.1:8000

Find your APK¶

Theoretically, you'll be running this against an APK you've built and developed. In that case, you'll have the APK handy and can load it into the UI. I, unfortunately, didn't have that luxury. Fortunately, getting the APK is relatively simple.

Go to the Google Playstore and search for the application
Navigate into the details page of the application (you should see the option to install it)
Copy the URL to this page
Go to APKCombo
Paste the URL you copied in step 3 into the appropriate text box and click "Generate Download Link"
After a few seconds, click the download icon next to the link that was generated

You now have the APK to use.

Using MobSF¶

With the APK in hand, and MobSF running, navigate to http://127.0.0.1:8000. Upload the APK and wait a couple of minutes. If you are watching the logs, you'll see it performing multiple steps behind the scenes. Once it's done analyzing the APK, it'll drop you to the dashboard.

The dashboard has a lot of detail about the APK. Scroll through to see the permissions it's utilizing, certificates attached to it, potential security vulnerabilities, code analysis, detected URLs in the code, hard coded secrets, and strings. All of these would help a team build a better and more secure mobile application.

Conclusion¶

For my purpose, this 15 minute exercise not only answered the question I was seeking it also raised a few items that will need to be corrected before the next release. Introducing security practices with a tool like MobSF would be incredibly beneficial to a team tasked with building an Android application. Further, adding in the dynamic analysis capabilities can provide more details on areas to improve.

While I didn't need those to answer my question, I'd encourage someone looking to introduce this to a team, to look at the capabilities of MobSF

Review of Data Analysis with Polars Udemy course

2024-01-29T11:30:00-06:00

Introduction¶

After the previous course about Polars, I was excited to learn more. I settled on Data Analysis with Polars with Liam Brannigan instructing. I was drawn to this because of the mention of visualization using Matplotlib, Seaborn, Altair and Plotly. The course is billed at 2.5 hours long and is currently going for $85, though I did pick this up during one of Udemy's many sales.

Let's get started on the review and why I rated it so low.

Course¶

I have multiple complaints about this course.

The entire course is reading from pre-built notebooks or slides.
The last 26 out of 27 lectures are one page slides.
The quizzes in the middle of the courses are not effective, because the course is not designed for the learner to actually do any coding.
The exercises at the end of lectures are never revisited.

This course felt like the instructor was given a bunch of Jupyter notebooks by a more senior instructor and told to go teach the class. There is a lot of reading directly from the notebook, quickly glossing over code, and then talking through the output of the code. Unlike the last course, this one doesn't even feel like a cookbook of usefulness. It's just a glorified slide deck.

Let me show you what I mean. This is the course list's last 4 sections. I've collapsed them to show only the section header so it's a reasonable sized screenshot.

There are 27 lectures in these 4 sections. The total estimated time for these 27 lectures - 7 minutes. There is only 1 video in this entire group of sections. The other 26 out of 27 lectures are one page slides. The slides are "What you'll learn by the end of this lecture" slides and a link to a Jupyter notebook.

The quizzes scattered through out the course are also ineffective. Many of the questions asked are asking what code provided is correct. These can be helpful, if the learner has done any hands on work during the course. But, this course isn't designed that way. It's an instructor reading slides, hand waving at some code, and then executing the code to show it works. It's ineffective.

The same problem holds true with the exercises at the end of many lectures. They are designed to get the student more experience and I started doing them early in the course. But, they are never revisited. The instructor doesn't talk about them. It's just homework that's assigned as busy work, essentially.

Conclusion¶

Avoid this course. Nearly half the course is just a single slide for the lectures. The items I was interested in learning about - visualizations - were among those one page slides. The course is designed like a glorified README, and its not worth the sales price I paid, let alone the full price.

Review of Learn Data Analytics with Polars (Python) Course

2024-01-25T08:30:00-06:00

Introduction¶

I started this entire Course Reviews series 7 years ago, with a post about a Pandas course. It went so well, I continued both taking various courses to expand my own knowledge and sharing my experiences with those courses here. It seems fitting to look at a Pandas competitors: Polars.

I've used Pandas for years and it's done the job well enough. But, Polars is gaining traction in the industry and one of the data engineers at work mentioned it as a possible tool to look at. I like to be informed when my engineers make tooling recommendations, so I took it upon myself to learn a little more about Polars. I did so by selecting a course by Kieran Keene, titled Learn Data Analytics with Polars (Python) in just 2 hours!. I got this course during another of Udemy's sales for $13.

About the course¶

The entire course is run out of Google Colab. This is nice, because you don't need to set up a virtual environment or install anything to take this course.

The course is taught with Polars version 0.17.3, which is from April 2023. I took this course in January 2024, so I used the current version - 0.20.2. This introduces a few very minor deprecations from the older version, but simply reading the deprecation warning for each tells you how to solve the problem.

Namely, the two deprecations that I recall running into:

In 0.17 there is the df.apply() function. This has been deprecated in favor of df.map_rows()
The groupby() function has been deprecated in favor of group_by()

Both take exactly the same arguments, so it was as simple as renaming the instructor's function to the modern one.

The one other minor difference is that the example data has changed at some point between when the course material was recorded and when I took the course. This doesn't change how any of the lectures behave or any of the examples act. It does change a couple results, so I couldn't compare exact numbers between what I received and what the instructor received.

This was not a problem though. The instructor does a very good job of explaining what each lecture is going to teach, shows at least one method of accomplishing the task, and then summarizing the lecture. Through this, it's easy to determine if the results you get with the different sample data returns an accurate result.

I enjoyed the short lectures. I believe the longest single lecture is 11 minutes long, but each builds off of what you've done previously. By introducing these natural break points, it's easy to complete a topic and then spend a couple minutes experimenting with other methods or logic to see how the library behaves.

Additionally, there are multiple ways to perform certain tasks, and the instructor takes the time to explain each of these. Building off of one another, it's easy to determine which is appropriate for the scenario being discussed.

Speaking of scenarios, the course concludes with two short challenges. The goal is to use the knowledge gained from previous lectures to figure out results to two problems. If you followed along through the hour and a half course, these should be pretty easy to figure out but do require combining multiple steps to get to the answer.

Conclusion¶

I enjoyed this course, the instructor's teaching method, and two challenges at the end to tie the course back to the lectures. This was a very hands on course and the breaks between lectures and sections encouraged experimenting with the library.

Experimenting helps me learn more about the tool, and I commend the instructor for building the course in such a way that students could try out the library.

I also found that I liked the Polars library. Pandas has its quirks and when I dig more into Polars, I'm sure I'll find it has some too. But, Polars feels easier to grasp. The API is easier to understand as you are reading through the code.

I think I'm going to try out another course to see what else the library can do. That alone should be a compliment to both the library and this course - it's encouraged me to keep learning more.

Review of Learn Github Actions for CI/CD DevOps Pipelines Course

2024-01-22T10:30:00-06:00

Introduction¶

I received a coupon code for Houssem Dellai's Learn Github Actions for CI/CD DevOps Pipelines Course. I've written about GitLab runners and the CI/CD process of that tool in the past. I've used GitHub actions to check this blog for broken links. I've used TravisCI (and written why not to trust TravisCI). I've also used BitBucket Pipelines in my professional life. I wanted to learn a bit more about what GitHub Actions could do and the coupon code made this 4 hour course worth the time to investigate.

You'll see below in more detail, but this course was not what I expected. It's advertised as being for "All beginners" and "Ops experts". I argue that both of these are incorrect. This course felt more like a 20-30 lecture "GitHub Actions Cookbook" than a course looking to explain GitHub actions.

Let's go through it...

Course Overview¶

There are multiple formats the instructor uses in this course. It's pretty obvious that the sections in this lecture were recorded independently. Some lectures take place entirely within the GitHub interface, some are using Powerpoint slides to drive the lecture, one is done on a whiteboard. To me, it seems the various lectures have been bundled together.

After a bit of research - and a mid-lecture "Check out my YouTube" - it's clear why that feeling exists. The lectures are available on YouTube. Not all of the videos on YouTube are part of this course, but scrolling through, I can quickly see that many of the videos in the course are on YouTube.

As for the content of the course, it doesn't match what I was expecting. With the "All beginners" advertisement, I was expecting more discussion and coverage on why certain things were being done. A discussion on the trade offs or the instructor's thought process on how to accomplish tasks.

Instead, the content is a lot of "this is my code to accomplish this task". It is a great course to get the quick run down of how to accomplish tasks, especially with Azure. However, it is not a good course if you want to know why a specific block of code is the best route to go. It's a cookbook to accomplish a task. A handful of lectures do cover theory, but these are few and very far between.

Conclusion¶

If my coupon hadn't knocked off the entire price of the course, I'd be upset with this one. I also picked up two other courses by this instructor and I suspect I'm going to find similar concerns with those courses. Honestly, that's too bad, because the content is good. It's just not marketed how it's packaged. With my expectations set for learning a bit, it's disappointing to get lectures that are designed to be copy and pasted from.

Don't come into this expecting to learn a lot of reasoning behind how things are accomplished. If you need an overview of how to accomplish the tasks in the content list though, you'll find that. It'll be a relatively dry walkthrough of the exact code you need. But, if you want to know why...this isn't the course for you.

If you can find a coupon code for this, I think it's worth your time to see how some of these advanced tasks are accomplished. If you can't find that coupon though, head over to the instructor's YouTube page and watch that section for free.

Python Gotcha: Modifying a list while iterating

2024-01-03T15:00:00-06:00

Introduction¶

A fairly common task an engineer may need to do is to remove all numbers from a list below a threshold. It's a simple enough task that anyone could accomplish - iterate over the list, check the value, if it's below the threshold remove it. With Python you can do this in a for loop.

Gotcha¶

Let's check code for this simple problem.

numbers_list = [0, 9, 2, 1, 5, 1, 10, 4, 3]
for index, value in enumerate(numbers_list):
    print(index, value, value < 5, 'Deleted' if value < 5 else 'Kept')
    if value < 5:
        del(numbers_list[index])

print(numbers_list)

It'd be reasonable to expect a final list that looks like this:

[9, 5, 10]

But, the actual output is:

[9, 1, 5, 10, 3]

What is happening?

Explanation¶

The (unconscious?) assumption that the developer is making with this piece of code is that the final list must be in numbers_list. Since we're able to iterate and modify, there is no need for a temporary variable or a need to worry about list copy gotchas. But, let's go through this one iteration at a time.

Starting list: [0, 9, 2, 1, 5, 1, 10, 4, 3]

| Iteration | Index | Value | Value < 5? | Action | List after iteration      |
| --------- | ----- | ----- | ---------- | ------ | ------------------------- |
| 0         | 0     | 0     | True       | Delete | [9, 2, 1, 5, 1, 10, 4, 3] |
| 1         | 1     | 2     | True       | Delete | [9, 1, 5, 1, 10, 4, 3]    |
| 2         | 2     | 5     | False      | Keep   | [9, 1, 5, 1, 10, 4, 3]    |
| 3         | 3     | 1     | True       | Delete | [9, 1, 5, 10, 4, 3]       |
| 4         | 4     | 4     | True       | Delete | [9, 1, 5, 10, 3]          |

End of iteration as list has no more elements

Looking at this, it's a little clearer what's going on. Let's go through these one iteration at a time.

On the first iteration, the value of Index 0 is 0, so it's deleted. And the problem starts now.
On the next iteration, the value of Index 1 is 2 (not 9). In the previous iteration, Index 0 was deleted, which means everything has moved up one index. The value of 9 was never checked. Instead, we look at a value of 2. It's less than our threshold so it's also deleted.
On the third iteration, the value of Index 2 is 5. Again, we've skipped a value (1) and it's never even checked. This iteration doesn't result in a deletion because 5 is not less than 5.
On the fourth iteration, the value of Index 3 is 1. This is deleted as expected.
On the fifth iteration, the value of Index 4 is 4. We've skipped another value (10) and this iteration's value is less than our threshold so it's deleted.
There is no further iteration, because the last one has made the final value the last index, so the final value is never checked.

We skipped checking several values because we messed with indexes that were either being operated on or hadn't been operated on yet. This results in unexpected behavior.

Solution¶

A quick solution is to add the values that exceed our threshold to another variable:

numbers_list = [0, 9, 2, 1, 5, 1, 10, 4, 3]
larger_list = []
for index, value in enumerate(numbers_list):
    if value >= 5:
        larger_list.append(value)

print(larger_list)

This results in the expected list of [9, 5, 10]

Another solution, assuming that the result needs to sit in a variable with the same name, is to use a list comprehension. This does the same thing as above, but keeps the code more compact. It also works because the final result is assigned to a new variable object with the same name.

numbers_list = [0, 9, 2, 1, 5, 1, 10, 4, 3]
numbers_list = [nums for nums in numbers_list if nums >= 5]
print(numbers_list)

This results in the expected list of [9, 5, 10]

Python Gotcha: List Copy Problems

2023-12-20T15:30:00-06:00

Introduction¶

This problem isn't limited to only lists. Any mutable object in Python can be impacted by this "gotcha". The problem is setting one mutable object equal to another doesn't make a copy. It assigns a new variable to the same object as the original. Let's take a look and see how this appears.

The First Gotcha¶

The first example is pretty simple. Let's make a copy of our list

>>> list1 = [1,2,3,4,5]
>>> list2 = list1
>>> list1
[1, 2, 3, 4, 5]
>>> list2
[1, 2, 3, 4, 5]

That looks like I have two lists, both with the same set of values. But, let's modify the second list and change the first element in the list to be 6

>>> list2
[6, 2, 3, 4, 5]

But what about list1? It also has changed.

>>> list1
[6, 2, 3, 4, 5]

This is occurring because, as mentioned, list2 is assigned to the same object as list1. Any change to either, changes the object that both are assigned to.

>>> print(f"list1 id: {id(list1)}")
list1 id: 1497534800448
>>> print(f"list2 id: {id(list2)}")
list2 id: 1497534800448

The First Solution¶

This can be solved a few different ways. The important take away, though, is that you have to create a new object with the original values. I can think of a few ways to handle this. This article doesn't consider performance for these basic lists.

One option is to utilize the list() method and pass it the initial values from list1

>>> list3 = list(list1)
>>> list3
[6, 2, 3, 4, 5]
>>> list3[0] = 0
>>> list3
[0, 2, 3, 4, 5]
>>> list1
[6, 2, 3, 4, 5]
>>> print(f"list1 id: {id(list1)}")
list1 id: 1497534800448
>>> print(f"list3 id: {id(list3)}")
list3 id: 1497531282368

A second option is to slice() the entire original list into a new variable.

>>> list4 = list1[:]
>>> list4
[6, 2, 3, 4, 5]
>>> list4[0] = 0
>>> list4
[0, 2, 3, 4, 5]
>>> list1
[6, 2, 3, 4, 5]
>>> print(f"list1 id: {id(list1)}")
list1 id: 1497534800448
>>> print(f"list4 id: {id(list4)}")
list4 id: 1497534798208

Lastly, the list object has a copy() method that you can utilize.

>>> list5 = list1.copy()
>>> list5[0] = 0
>>> list5
[0, 2, 3, 4, 5]
>>> list1
[6, 2, 3, 4, 5]
>>> print(f"list1 id: {id(list1)}")
list1 id: 1497534800448
>>> print(f"list5 id: {id(list5)}")
list5 id: 1497534799552

With each of these three options, you can see that we created new objects because each has a unique object id. Changing one of these copies does not change the original's values.

The Second Gotcha¶

You, a savvy developer, know all this though. You also have a more advanced data structure. You have a list of lists.

[
    ['first element', 1, 2, 3],
    ['second element', 4, 5, 6],
    ['third element', 7, 8, 9],
]

Since you know about this gotcha, you are going to make a new object and copy the values to the new object.

>>> deeplist1 = [
...     ['first element', 1, 2, 3],
...     ['second element', 4, 5, 6],
...     ['third element', 7, 8, 9],
... ]
>>> deeplist2 = deeplist1.copy()
>>> print(f"deeplist1 id: {id(deeplist1)}")
deeplist1 id: 1497534571008
>>> print(f"deeplist2 id: {id(deeplist2)}")
deeplist2 id: 1497534577536

Now, with two different objects, is it safe to modify an element in one of the sublists? Turns out, it is not.

>>> deeplist2[0][0] = "NOT 3RD"
>>> deeplist2
[['NOT 3RD', 1, 2, 3], ['second element', 4, 5, 6], ['third element', 7, 8, 9]]
>>> deeplist1
[['NOT 3RD', 1, 2, 3], ['second element', 4, 5, 6], ['third element', 7, 8, 9]]

But why? The ids of deeplist1 and deeplist2 are different. This is true, but the sublists are not. Each sublist is assigned the same object id as the original.

>>> id(deeplist1[0][0]), id(deeplist2[0][0])
(1497534796864, 1497534796864)

The Second Solution¶

Utilize the deepcopy() method to make a copy of your original object.

>>> import copy
>>> deeplist3 = copy.deepcopy(deeplist1)
>>> deeplist3[0][0] = "NEW VALUE"
>>> deeplist3
[['NEW VALUE', 1, 2, 3], ['second element', 4, 5, 6], ['third element', 7, 8, 9]]
>>> deeplist1
[['NOT 3RD', 1, 2, 3], ['second element', 4, 5, 6], ['third element', 7, 8, 9]]

Using deepcopy() we can safely make a copy of our list of lists and modify both the original and the copy without impacting the other.

Python Gotcha: Timezone Naive Functions like utcnow() and utcfromtimestamp()

2023-11-23T10:15:00-06:00

Introduction¶

There are two common functions that are naive datetime objects. The idea behind these types of objects is that timezones don't matter. Perhaps you always assume that datetimes are in a set timezone. Or, you are aware of timezones and want to make things easy on yourself so always assume datetimes are set to UTC.

This is your mistake.

The utcnow() and utcfromtimestamp() functions are naive, but other datetime functions are not.

The Gotcha¶

>>> from datetime import datetime
>>> dt = datetime.utcfromtimestamp(0)
>>> dt
datetime.datetime(1970, 1, 1, 0, 0)
>>> dt.timestamp()
21600.0

What happened here?

This was run on a computer in the Central Standard Time time zone. First, I created an object set to the UNIX epoch - Jan. 1, 1970. Since I utilized utcfromtimestamp() to do this, there is no timezone data attached to this object. When I use the standard timestamp() function, Python sees there is no timestamp to utilize, so it uses the system configured timezone. Hence the 21,600 second offset from the epoch.

This didn't always happen though. Sometime between Python 2 and Python 3 this behavior changed. This assumption that "no timezone means local timezone" introduced this gotcha. In my opinion this is a rather dangerous one too.

Solution¶

Utilize explicit time stamps everywhere. From The Zen of Python - PEP 20:

Explicit is better than implicit.

>>> import datetime
>>> dt = datetime.datetime.fromtimestamp(0, datetime.UTC)
>>> dt
datetime.datetime(1970, 1, 1, 0, 0, tzinfo=datetime.timezone.utc)
>>> dt.timestamp()
0.0

Notice that by adding datetime.UTC to my fromtimestamp() call, I get a datetime object with tzinfo. This makes the conversion from the datetime object back into a timestamp safe.

This is the way that Python is moving. In an effort to correct these unsafe assumptions that were introduced early in Python 3, the naive datetime functions have been deprecated. If you were to run my first block of code in Python 3.12, you receive a deprecation warning.

>>> dt = datetime.utcfromtimestamp(0)
<stdin>:1: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).

Notice that the message is telling you to add the timezone information and to utilize fromtimestamp(). In the coming year - years? - both the commonly seen utcnow() and utcfromtimestamp() will be removed.

In my opinion, as someone that has fought this bug in the past, it's a welcome change. Timezone assumptions are difficult to troubleshoot. Assumptions about underlying behavior make it a challenge. Running the same script in different timezones, like you would with a geographically diverse team, makes it even harder. It's especially annoying if one of your coworkers is utilizing UTC as their time zone so that the script behaves properly for them but no others.

I have code bases that will need to make this change, but in the end, they will be easier to maintain with the removal of the utc* functions.

Ruby Gotcha: Operator Precedence

2023-10-23T10:45:00-05:00

Introduction¶

In one of my previous roles, the team utilized Ruby. I was not an individual contributor at that point in my career, so my exposure to it was less than much of my team. However, during my tenure, I noticed several "Gotchas" of the Ruby language. Each language has these in their own way, but in this post I'm going to cover one specific one.

What's the difference between and and &&? What about the difference between or and ||?

The Gotcha¶

At first pass, these two blocks of code appear to do the same thing. Let's take a look.

irb(main):003:0> true && false
=> false
irb(main):004:0> true and false
=> false

By the rules of boolean logic, both and and && statements return false.

irb(main):005:0> true || false
=> true
irb(main):006:0> true or false
=> true

Boolean logic holds true for the or and || statements and return true.

But, what happens if we assign the result of this to a variable?

irb(main):007:0> result = true and false
=> false
irb(main):008:0> result1 = true && false
=> false
irb(main):009:0> result
=> true
irb(main):010:0> result1
=> false

Now we see the problem. With the and operator, result = true. With the && operator, result1 = false.

Looking at the or/|| operators quickly,

irb(main):011:0> result2 = true or false
=> true
irb(main):012:0> result3 = true || false
=> true
irb(main):013:0> result2
=> true
irb(main):014:0> result3
=> true

This all looks reasonable. But, let's swap the order of our true/false in the statement.

irb(main):015:0> result2 = false or true
=> true
irb(main):016:0> result3 = false || true
=> true
irb(main):017:0> result2
=> false
irb(main):018:0> result3
=> true

Here we see the differences again, with the or operator setting result2 = false and the || operator setting result3 = true

What is going on?¶

and and && / or and || are the same thing, but they have different orders of precedence. The (not well formatted in my opinion) documentation shows the precedence table for Ruby.

The key things to point out here:

&& and || are above and and or, meaning they will be evaluated first.
= is also above and and or (but below && and ||)

Let's look at one of the examples again:

irb(main):015:0> result2 = false or true
=> true
irb(main):017:0> result2
=> false

With the order of precedence in mind, this is what happens:

result2 is assigned the value of false because = occurs before or
The remainder of the statement is evaluated as true

This was hidden when true and false was swapped because result2 was assigned a value of true and then the remainder of the statement was also evaluated to true.

How do you prevent this?¶

There are very few reasons to utilize the English words and and or in Ruby's logical evaluations. Instead, these should be utilized as flow control modifiers (think if and unless) and not for boolean logic.

Here's an example of how it should be utilized for flow control

irb(main):019:0> numeral = 10 && numeral / 5
(irb):19:in `<main>': undefined method `/' for nil:NilClass (NoMethodError)
        from C:/Ruby32-x64/lib/ruby/gems/3.2.0/gems/irb-1.6.2/exe/irb:11:in `<top (required)>'
        from C:/Ruby32-x64/bin/irb:33:in `load'
        from C:/Ruby32-x64/bin/irb:33:in `<main>'
irb(main):020:0> numeral = 10 and numeral / 5
=> 2

In our first example, with &&, the statement is broken up like this

numeral = (10 && numeral) / 5

This won't work because numeral doesn't have a value when && is evaluated. Contrast this with the second version where the code:

Assigns a value of 10 to numeral
Divides numeral by 5

While basic, it should get the point across. Another example could be something like this:

user = User.find_by_email(email) and user.send_email!

Here, we are only sending an email to the user if we find their information. No information, no email is sent.

Python Gotcha: Comparisons

2023-10-18T10:15:00-05:00

Float Equality Comparisons¶

Like every other programming language, Python can't accurately represent floating-point numbers. I'm sure many Computer Science students have lost many hours learning how floating representation works. I remember that class well.

In any case, let's get into the problem with comparing float values in Python and how to handle it.

>>> 0.1 + 0.2 == 0.3
False

You, me, and anyone with a few years of elementary school under their belt can see that this should be True.

What's happening here? We can see the problem by breaking down the component parts of this comparison.

>>> 0.3
0.3
>>> 0.1 + 0.2
0.30000000000000004

And now we see the floating point representation that is causing a problem.

So, how do we deal with this?

Decimal¶

There are a couple options, both of which have their drawbacks. Let's start with Decimal.

The decimal module provides support for fast correctly rounded decimal floating point arithmetic.

This sounds good, but an important gotcha here too is how it handles numerals vs. strings.

>>> from decimal import Decimal
>>> Decimal(0.1)
Decimal('0.1000000000000000055511151231257827021181583404541015625')
>>> Decimal('0.1')
Decimal('0.1')

This means, to accomplish the comparison above, we need to wrap each of our numbers in a string.

>>> Decimal('0.1') + Decimal('0.2') == Decimal('0.3')
True

That's annoying, but does function.

isclose¶

Python 3.5 implemented PEP 485 to test for approximate equality. This is done in the isclose function.

>>> import math
>>> math.isclose(0.1+0.2,0.3)
True

That's cleaner than wrapping everything in strings. But, it also is more verbose than a simple == statement. It makes your code less clean, but does provide accurate comparisons.

is vs. ==¶

Another comparison that I've commonly seen misapplied is developers using is when they mean ==. Put simply, is should ONLY be used if you are checking if two references refer to the same object. == is used to compare value by calling underlying __eq__ methods.

Let's see this in action:

>>> a = [1, 2, 3]
>>> b = a
>>> c = [1, 2, 3]
>>> d = [3, 2, 1]
>>> a == b
True
>>> a == c
True
>>> a == d
False

So far, nothing unusual. a has the same values of b and c and different values from d. Now let's use is

>>> a is b
True
>>> b is a
True
>>> a is c
False
>>> a is d
False
>>> b is c
False

Here, the only True statements are the comparison between a and b. This is because b was initialized with the statement b = a. The other two variables were initialized as their own statements and values. Remember, is compares object references. If they are the same, it returned True.

>>> id(a), id(b), id(c), id(d)
(2267170738432, 2267170738432, 2267170545600, 2267170359040)

Since a and b are the same object, we get a True on their comparison. The others are different, hence the False.

nan == nan¶

nan, or Not a Number, is a floating point value that can not be converted to anything other than a float and is considered not equal to all other values. It's a common way to represent missing values in a data set.

There is a key phrase in that description above that is the basis for this Gotcha:

is considered not equal to all other values

It's common for software to check if two values are equal to one another prior to taking an action. For nan, that does not work:

>>> float('nan') == float('nan')
False

This prevents code like this from entering the if block of an if/else statement

>>> a = float('nan')
>>> b = float('nan')
>>> if a == b:
...   .. ## Do something if equal
... else:
...   .. ## Do something if not equal

In this example, they are never equal.

This leads to an interesting, if not unintuitive, way of checking if a variable is a nan value. Since nan is not equal to all other values, it is not equal to itself.

>>> a != a
True

Like I said, "Interesting". But, when your code is looked at by others it's also "confusing". There is an easier way to show that you are checking for a nan value. isnan

>>> import math
>>> a = float('nan')
>>> b = 5
>>> c = float('infinity')
>>> math.isnan(a)
True
>>> math.isnan(b)
False
>>> math.isnan(c)
False

To me, that's a much clearer check that we want to see if the value is nan. It's likely you aren't just passing nan to a single variable. You're probably using a library like NumPy or Pandas. In that case, you have functions in each of those libraries that can check for nan in a performant way.

In NumPy the function has the same name but in the NumPy library: numpy.isnan(value).
In Pandas, the function has a slightly different name: pandas.isna(value)

Conclusion¶

Comparisons aren't always as straight forward as we'd like. I covered a few common comparison problems in Python here.

Floating point comparisons are common across languages. Python has a few ways of making this easier for developers. I recommend utilizing isclose as it keeps the code a bit cleaner and eliminates the need to wrap numbers in strings if using the Decimal module.

is should only be used to check if two items are referring to the same object. In any other case, it's not doing the check you want it to be doing.

Lastly, nan is equal to nothing else. It's important to be aware of that before you start comparing values in your dataset to one another.