Andrew Wegner | Ponderings of an Andy - Technical Solutions

Python Gotcha: Reusing Generators Returns Nothing

2025-07-22T09:00:00-05:00

Introduction¶

In the previous article, we looked at logging uncaught exceptions. Let's utilize the log output from that post for another common task: error log file processing. This example is going to be pretty simple, as the error log for the post is tiny, but in a production environment this could be hundreds of thousands of lines, or gigabytes in size. Potentially, that's a lot to shove into memory for processing. But that's where a generator can come in to help.

Generators do not store their results, instead they maintain state and yield the result back to the caller. This means each line in a log can be processed and returned, without loading the entire file.

Flashback¶

As a reminder from the previous article, the log file being used looks like this:

2025-07-14 22:30:44,061 __main__ INFO Application start
2025-07-14 22:30:44,061 __main__ CRITICAL uncaught exception, application will terminate.
Traceback (most recent call last):
File "/home/andy/main.py", line 31, in <module>
    main()
    ~~~~^^
File "/home/andy/main.py", line 27, in main
    logger.info(divide(a,b))
                ~~~~~~^^^^^
File "/home/andy/main.py", line 21, in divide
    return a/b
        ~^~
ZeroDivisionError: division by zero

There is one INFO entry and one CRITICAL entry.

yield vs return¶

It's important to understand what makes a generator different from a regular function. The key distinction is the yield keyword. When a function contains yield, Python treats it as a generator function, which behaves differently from functions that use return.

A regular function with return executes completely, returns back a single result and then terminates. A function with yield creates a generator object that can pause execution, return a value, and later resume from exactly where it left off. This is what enables the memory-efficient and lazy evaluation that makes generators powerful.

Gotcha¶

Log processing¶

A generator result can only be utilized one time. Whether you are using a generator to output the next item in a sequence or process a file line by line, once you have passed an iterable or exhausted the generator, it doesn't get reused. This simple generator demonstrates the issue.

def read_log_lines(filename):
    with open(filename, 'r') as f:
        for line in f:
            if 'CRITICAL' in line:
                yield line.strip()

error_logs = read_log_lines('app.log')

error_count = len(list(error_logs))
print(f"Found {error_count} CRITICAL lines")

recent_errors = [log for log in error_logs if '2025' in log]
print(f"Recent errors: {len(recent_errors)}")

At first glance, it looks like this will read the log file, count the number of errors and then output how many of those were in 2025 (or contain the string 2025). However, the actual output is different.

Found 1 CRITICAL lines
Recent errors: 0

error_logs is a generator object. If a function yields, it is a generator. As error_count is initialized, it processes the error log and yields back any critical lines. The list() function will consume the entire generator (file). A few lines later, the developer wants to see how many of these are recent errors and attempts to go through the error_logs generator again. Success! No recent errors!

Right?

No, and looking at the log quickly shows that.

Fibonacci¶

Let's use a generator to build the Fibonocci sequence. Spoiler for interviews! In this case, I'm going to use a generator to get the first 10 items. Then print out the first 5 and then try to print the entire list of 10 items.

def fibonacci(n):
    a, b = 0, 1
    for _ in range(n):
        yield a
        a, b = b, a + b

fib_numbers = fibonacci(10)

print("First 5 numbers:")
for i, num in enumerate(fib_numbers):
    print(num)
    if i >= 4:
        break

print("Entire List:")
full_list = list(fib_numbers)
print(full_list)

The output for this is:

First 5 numbers:
0
1
1
2
3
Entire List:
[5, 8, 13, 21, 34]

Notice that the full_list variable only contains the items remaining on the generator. Since the first 5 (indexes 0 through 4) were printed, they are no longer part of the generator. When the full list is printed, only the remaining items can be printed.

Solution¶

The solution to the problem is easy enough. Call the generator function again. For example, with the log code from above:

error_logs = read_log_lines('app.log')
error_count = len(list(error_logs))
...
error_logs = read_log_lines('app.log')  # Call again and create a new generator
recent_errors = [log for log in error_logs if '2025' in log]

For the Fibonacci code, you would call fib_numbers = fibonacci(10) again before printing the full list.

Obviously, there is a down side here with duplicate processing of the same data due to running the generator twice. This could probably be solved with some logic adjustments to the generator or the code calling the generator, but that'll vary by application depending on what the generator is doing.

Conclusion¶

The important thing to take away from this is that once you have iterated over an item in a generator, it's no longer part of a generator. This means that if you want to get clever and see if there are more items in a generator, or determine the next item, you've consumed the next item.

The power of generators, especially when processing large amounts of data, can't be understated. But, at the same time, it's important to know that reusing an exhausted generator or attempting to access a previous item directly from the generator is not going to work. Instead, to reuse generator logic, call the generator function again to create a new generator object, or convert to a list if memory permits and multiple iterations are needed.

Python Gotcha: Logging an uncaught exception

2025-07-14T23:00:00-05:00

Introduction¶

A well build application will use logging instead of print statements. An exceptionally well built one will log in such a way that additional context is added to each log message and be consumable by a log aggregation service. Perhaps I'll write up such an article in the future. For now though, let's focus on a single problem.

Here is some sample code to demonstrate the problem.

import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

handler = logging.FileHandler("app.log")
formatter = logging.Formatter("%(asctime)s %(name)s %(levelname)s %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)

def divide(a, b):
    return a/b

def main():
    logger.info("Application start")
    a = 10
    b = 0
    logger.info(divide(a,b))
    logger.info("Application end")

if __name__ == "__main__":
    main()

Briefly, this sets up the app.log file to receive our log messages. The main function is going to divide two numbers, log the result, and end the program. Pretty simple.

Except, in this case, it is dividing by zero. This throws an error and crashes the program.

The console spits out a stack trace

Traceback (most recent call last):
File "/home/andy/main.py", line 22, in <module>
    main()
    ~~~~^^
File "/home/andy/main.py", line 18, in main
    logger.info(divide(a,b))
                ~~~~~~^^^^^
File "/home/andy/main.py", line 12, in divide
    return a/b
        ~^~
ZeroDivisionError: division by zero

The app.log file contains a single line:

2025-07-14 22:20:04,551 __main__ INFO Application start

Gotcha¶

Where is the gotcha here? The stack trace is right there!

You are, of course, right. However, imagine that this was not a simple application, but instead a production application that sends logs to a central service. Your application crashed and now one was watching the console. Your app.log file has no information. It says the application started and then...nothing. What happened? Is it still running?

As you dig through running processes, or check a /health end point for responses, you find out that it isn't running. That took a lot of time, and production isn't responding.

You've lost all visibility to what happened in your application at the most critical moment. When it crashed and spit out a stack trace, you want as much detail as you can get.

Solution¶

The solution is sys.excepthook. This is called when any exception is raised and uncaught, except for SystemExit. It's pretty easy to utilize as well. A few small changes to the above code will allow us to log this completely unexpected ZeroDivisionError.

import logging
import sys

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

handler = logging.FileHandler("app.log")
formatter = logging.Formatter("%(asctime)s %(name)s %(levelname)s %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)

def handle_uncaught_exception(exc_type, exc_value, exc_traceback):
    logger.critical(
        "uncaught exception, application will terminate.",
        exc_info=(exc_type, exc_value, exc_traceback),
    )
sys.excepthook = handle_uncaught_exception


def divide(a, b):
    return a/b

def main():
    logger.info("Application start")
    a = 10
    b = 0
    logger.info(divide(a,b))
    logger.info("Application end")

if __name__ == "__main__":
    main()

The important bit is the new handle_uncaught_exception function and the sys.excepthook line (with appropriate import statement).

Someone running this in the console will notice that there is not a stack trace dumped to the console now. Instead, our log contains important information:

2025-07-14 22:30:44,061 __main__ INFO Application start
2025-07-14 22:30:44,061 __main__ CRITICAL uncaught exception, application will terminate.
Traceback (most recent call last):
File "/home/andy/main.py", line 31, in <module>
    main()
    ~~~~^^
File "/home/andy/main.py", line 27, in main
    logger.info(divide(a,b))
                ~~~~~~^^^^^
File "/home/andy/main.py", line 21, in divide
    return a/b
        ~^~
ZeroDivisionError: division by zero

This is great! Now when troubleshooting this failing application and looking at the logs, we can easily see that an exception occurred. Additionally, with proper updates to the logging, more context can be provide such as the values of a and b. While it's easy enough to figure out that b is 0 in this simple example, the context in a larger production application could save a ton of troubleshooting time.

Conclusion¶

Logging is vital to knowing what your application is doing. But, it's even more important to determining why it stopped working. If your logs aren't providing information when the application crashes, it's functionally useless. By implementing an excepthook, you can catch and properly log uncaught exceptions.

I know some of you are coming up with alternatives. Terrible ideas like wrapping the entire main block in a try/except. There are legitimate reasons to throw an exception. In this case, a ZeroDivisionError is a great exception to catch. But, you'd want to do it around as small of a code block as possible.

This is a clean way to catch truly unexcepted exceptions, not something a developer could have anticipated and, perhaps, fixed with additional input validation.

How to fix GitLab 18 error regarding git_data_dirs

2025-07-06T08:00:00-05:00

Introduction¶

Several years ago I set up GitLab in my home environment. I apprecate past me documenting the basic steps I took to do this, because I'll need them again at some point when I make upgrades to my home lab. I've documented somethings I've done (like setting up GitLab runners, or dealing with Grafana being deprecated within GitLab or utilzing GitLab to automatically backup my Obsidian notes) Unfortunately, I didn't document everything. One of those things that I didn't document was changing where my repositories are stored on disk by default.

In GitLab 17.8 (January 2025), a new deprecation warning started appearing.

git_data_dirs has been deprecated since 17.8 and will be removed in 18.0. See https://docs.gitlab.com/omnibus/settings/configuration.html#migrating-from-git_data_dirs for migration instructions.

I'm a little annoyed that there were only 5 months of notice on this because major versions are released in May. In any case, if you attempt to update to version 18 or beyond and are utilizing git_data_dirs, the upgrade will fail.

Solution¶

As usual, GitLab is good at providing documentation on resoluting and migrating through the deprecations. However, on my first attempt I encountered an error. I believe it's because I missed a note buried in the text - not the code blocks - the first time.

Note that the /repositories suffix must be appended to the path because it was previously appended internally.

The solution is to open /etc/gitlab/gitlab.rb (you probably should make a backup first). Find the git_data_dirs line. For me this was around line 455. Then comment out this entire block.

Then find (or add) the gitaly['configuration'] block. This was immediately after the git_data_dirs section for me. Uncomment it, and add the appropriate path (from your git_data_dirs block) and add /repositories to the end of it.

Once you are done, run

gitlab-ctl reconfigure
gitlab-ctl restart

Give it a minute to fire up all the GitLab subcomponents, and then you should be able to update GitLab beyond version 18.

New code section¶

My gitlab.rb now has this:

#git_data_dirs({
#  "default" => {
#    "path" => "/previous/path/to/repos"
#   }
#})

### Gitaly settings
gitaly['configuration'] = {
storage: [
    {
    name: 'default',
    path: '/previous/path/to/repos/repositories',
    },
],
}

Conclusion¶

With this quick change, I can continue to hold the repositories at a location of my choosing. The biggest thing is to add /repositories to the path in the new Gitly configuration. With this change, I can continue to utilizing the current version of GitLab - a tool that I still find invaluable.

Python Gotcha: Identity vs Equality - When 'is' Fails Unexpectedly

2025-06-17T08:00:00-05:00

Introduction¶

In the comparisons gotcha I wrote a few years ago, I briefly touched on is vs ==.

Put simply, is should ONLY be used if you are checking if two references refer to the same object. Remember, is compares object references.

Even more simply, is is not checking value. Let's take a look at a couple examples.

Gotcha¶

Integers¶

Python, specifically CPython, caches the values of -5 through 256 (inclusive). This means that these small integer values will always refer to the same object.

Note the phrasing there - "the same object".

Outside of that range, though, the same is not true.

>>> a = 100
>>> b = 100
>>> a is b
True
>>> a = 257
>>> b = 257
>>> a is b
False

In both of the above examples, using a == b would have returned True. The mistake was assuming that is does the same thing. It does not.

Strings¶

String interning is a method of storing only one copy of each distinct immutable string value. Immutable strings can't be changed. Not every string will be interned though. Let's take a look:

>>> a = "Hello"
>>> b = "Hello"
>>> a is b
True
>>> a = "Hello World"
>>> b = "Hello World"
>>> a is b
False
>>> a = "Hello_World"
>>> b = "Hello_World"
>>> a is b
True

The first and last example interned the strings, showing that a and b refer to the same object. But, the second example - Hello World - didn't get interned, so a and b refer to different objects. Why is this?

The short and simply answer is that any string that has only numbers, letters or underscores will be interned. Since Hello World contains a space, it would not be interned.

The Solution¶

To a new developer that has seen tutorials that read if a is True or if b is None, a conditional for integers or strings following the same pattern appears to be comparing values. If they test it with small, positive numbers or simple one word strings, the assumption holds up.

But, == is for comparing values! Each of the above examples would return True by changing the statement to a == b.

The few times that is is appropriate are when you are checking True/False or None. Otherwise, the vast majority of the time, you want to use an equality check (==)

The Python PEP8 programming recommendations state:

Comparisons to singletons like None should always be done with is or is not, never the equality operators.

The linters in the Python ecosystem report on the usage of is vs == too. flake8 has E711 - Comparison to None should be 'cond is None:'. ruff has a similar report with it's None comparison check.

I highly recommend a linter for your projects to catch this, and other problems that go against best practices.

Remember, is compares object references, not object equality

Moving MS Authenticator to a new phone without a personal account

2024-11-27T12:15:00-06:00

Introduction¶

Work utilizes the Microsoft Authenticator app as their 2FA application of choice. It's not my favorite, but it does what it needs to do, and since I only use the application for work things, it's nice to keep work accounts out of personal accounts in my other 2FA application. Recently, I got a new phone, and needed to migrate everything over to that device. As an aside, Android continues to make this easier and easier.

Standard advice on how to migrate your token to a new device is to enable cloud backup, sign in to the new device and import the backup. The problem is when you go to enable Cloud Backup from the settings, you are hit with this message:

"You need a personal Microsoft account to use cloud backup"

Well, I don't have a personal Microsoft account.

Aside: I can't provide screenshots of the Android application for this post because the application prevents screenshots from being taken. I appreciate this feature as part of the application for security purposes, but it's annoying when trying to write about it. Oh well...

Alternative to Cloud Backup¶

I have set up MFA on my work account - obviously - and can access settings through the Microsoft MFA setup page. From here, you'll be on the Security Info page. It will show the current authenticator device being utilized. You'll need the old device during this process to authenticate one last time.

Click on "Add sign-in method" and then "Microsoft Authenticator"

On your phone, ensure you have the MS Authenticator application installed. Then follow the on screen prompts between your computer and your new phone. You'll select "Work or School" and then scan the QR code that is common with 2FA/MFA applications.

The last step in the process will be to send an authentication request to the new device. Do so, enter the verification code, and your new device will be added.

I recommend you wrap this up by removing your old device from the Security Info section now that the new device is approved.

Conclusion¶

The Microsoft authenticator application doesn't make moving to a new device as easy as some of its competitors. The requirement for a personal Microsoft account to go through the most commonly recommended way is also a non-starter if you don't have such an account already. Fortunately, the alternative posted above ended up working just fine. I assume I'll need a new phone again, and this should help at least me, remember that there is an easy enough way to migrate.

Format USB drive with EFI partition in Windows

2024-11-13T13:00:00-06:00

Introduction¶

I seem to collect USB drives. They are handed out at conferences. They are used as recovery media for new machines. They just randomly appear on my desk. I use them to build live Linux CDs, transferring media from one device to another, building recovery media, and generally the things you'd expect a USB drive to be used for. Generally, when I use a USB drive the first thing I do is format it. My USB drives are not for long term storage and I consider anything on them used one time then the drive goes back to the bin for reuse another time.

An annoyance with this, though, is that when I use a drive to build a bootable backup media, often times an EFI partition will be created. As soon as this is created, Windows prevents that partition from being deleted in the future without taking additional steps. I have to go look those up every time because I do it infrequently enough to remember. This is a simple walkthrough of my process to solve this problem

A USB drive with an EFI partition looks like this within the Disk Management tool in Windows

Format an EFI Partition¶

To remove this partition, utilize the built in diskpart tool. Win + R and type in diskpart and allow the application to run.

First you need to find which drive you want to adjust partitions on.

Attention

Ensure you select the correct disk. Failure to do so could result in an inoperable system

Plug the USB drive into the machine and run list disk to see all available disks.

DISKPART> list disk

Disk ###  Status         Size     Free     Dyn  Gpt
--------  -------------  -------  -------  ---  ---
Disk 0    Online         3726 GB  1024 KB        *
Disk 1    Online          931 GB  1024 KB        *
Disk 2    Online           14 GB    14 GB

You can see that I have 3 drives detected. Disk 2 is my USB drive.

Select the disk.

DISKPART> sel disk 2

Disk 2 is now the selected disk.

Then show the partitions on the disk.

DISKPART> list partition

Partition ###  Type              Size     Offset
-------------  ----------------  -------  -------
Partition 1    System            4000 KB   850 KB

There is a single partition - the EFI partition. The rest of the drive is unallocated, as the screenshot above shows. Select this partition. IF you have multiple partitions listed, you'll need to determine which is the EFI partition. In my case, since I'm going to be formatting the entire drive anyway, if there were multiple listed I'd select and delete each individual partition.

DISKPART> sel partition 1

Partition 1 is now the selected partition.

Attention

These commands will delete a partition. If you haven't selected the right drive or partition things will break.

Finally, delete the EFI partition

DISKPART> delete partition override
DiskPart successfully deleted the selected partition.

From the disk management tool, you can verify it's gone and format as normal.

Controlling power to LED lights with WLED Controller and a Relay switch

2024-08-16T12:00:00-05:00

Introduction¶

It's time to continue my summer project of setting up some outdoor LED lights. Previously, I set up and updated the Ericsity controller with WLED.

I am putting up approximiately 20 meters - 65 feet - of LEDs. Total this will be nearly 2,000 individual LEDs and approximately 665 individually controlled LED segments (3 LEDs per segment). This will take more power than the little controller can handle. To demonstrate the problem of powering all of these LEDs with only the controller, look at this image:

These strips are wired with the end of one strip connected to the start of the next. The top strip is connected directly to the controller. The one below is the end of strip two, followed by the start of strip three and the bottom is the very end of the full run of LEDs. The lights are all set to the same color, but as you can see they clearly aren't the same color. The voltage drop across 65 feet of LEDs means that the LEDs are the end can't get enough power to match their earlier siblings.

Power Injection¶

The solution to this specific problem is to inject power into the LED strips. I picked up some WAGO connectors and a NUOFUWEI power supply. I also had some 18 guage wire on hand. With this power unit, I can easily set up three injection points.

The goal was to inject power at the start of strip 1, right where the controller is connecting in the image above. Then inject between strips 2 and 3, in the middle of the run. Finally, I injected power at the end of strip 4. With these three, equally spaced injection points, I was able to get a nice uniform color across the entire 20 meter run.

Problem Solved! Right?

Not exactly.

Turning off the PSU¶

WLED controls the lights, but doesn't control the power supply with this wiring setup. Turning off the lights within WLED does turn off the LEDs, but the power supply continues to run. It's not drawing at full load, but it is drawing power and the cooling fan is active. It's noticable and unneeded. I want a way to turn off the power supply AND the LEDs at the same time.

I can't power the WLED controller from the large power supply to do this, because if I turn off the power supply that'd also turn off the WLED controller. I'll need to power the WLED controller independently from the lights. Fortunately, this won't be a problem.

The next step is figuring out how I can use WLED to control the larger power supply. Fortunately, WLED has the ability to control a relay, which I can use to control the power supply. The Ericsity controller also has two output data pins. While I don't think the second one was built in to control a relay, it works perfectly here.

Wiring¶

I purchased a HiLetGo relay so that I could toggle the larger power supply on and off. To do this, it's important that the data line and the ground are common among the controller, the power supply and the LEDs.

The diagram above is a rough schematic of how I wired this. The controller sends data, but not power, to the LEDs. Data was on GPIO 16. The important part here is that the data line is shared across all of the strips. I did not need a signal booster for my project, and because I'm about to use the second exposed data channel for the relay, I had to ensure that this single channel could send a signal down the entire length of the strip. Fortunately, I didn't have any issues.

The wiring ground was tied into the PSU and the LEDs as well. The common ground is important.

On the other side of the diagram is the relay. I put this relay between the wall and the PSU. When WLED sent an ON signal, it would close the relay, turning on the PSU and the LEDs. GPIO 2 was tied to the relay and within LED Preferences, the Relay Pin was set to GPIO 2.

One quick power cycle from within WLED is required at this point. Press the power button in the UI to turn everything off, press it once more to turn it on. As long as the PSU is plugged into the wall, you should here it fire up and see the LEDs turn on. If you press the power button again, the LEDs turn off, the relay clicks, shuts down the PSU and only the WLED controller remains active.

Success¶

With the relay in place, the LEDs don't draw phantom power while off because the PSU isn't active. The added benefit, at least for this specific power unit, is the the fans aren't running constantly so it's not as loud. While this will eventually be outside, I'd still prefer to not hear the fan when the lights are off. While they are active it's not going to be bothering me, because I'll likely have music playing for the sound reactive features which will easily be louder than this fan.

The next step in the project is going to be to get this set up outside.

Setting up and updating the Ericsity WLED Controller from 0.13.3 to 0.14.1

2024-06-20T15:45:00-05:00

Introduction¶

My summer time project this year is to set up some LED lights outside. After testing out a couple Govee strips that I didn't like because I couldn't diffuse the individual lights away, I finally found an LED strip I liked. The next step was to control these strips so that I could do more than solid colors or the default rainbow every LED strip has. I settled on using the amazing WLED project to control the lights. To speed up the project, I decided to get a prebuilt controller and eventually selected the Ericsity controller with a built in mic.

Side note: This post is just about how I updated the controller. The next article in the post is available, and talks about how I am controlling a larger power supply with the WLED controller.

The Problem¶

The controller arrived and was easy to set up. I wired up my strips in the basement to ensure I didn't have any power problems and then determined where I needed to make cuts and solder points once this was outside. Then I went to set up segments in WLED so that I could control effects in specific areas.

The problem I quickly encountered was that I am outlining my deck with these lights and right in the middle of the deck is a set of steps that I also wanted to light up and then the rest of the deck rail. WLED doesn't have the ability (at least that I've found) to span an effect across multiple segments and I'd need multiple segments for this.

At minimium I'd need 3 segments - The rail, the steps, the rest of the rail. If I want an effect to span the entire rail, I can't do that with the steps in the middle. Further research pointed me to the ability to remap the physical LED order to a logical LED order. After trying to set this up, it wasn't feature complete in 0.13, but did appear to be in 0.14.

However, the Ericsity - rightfully so - utilizes a stable version of WLED that only allows you to update to 0.13.4 and the README has the following note:

This repository is still maintained, and will receive bugfixes. However no new features will be added.

Setting up Ericsity Controller¶

Let's take a step back and get the controller set up and then focus on upgrading it. Fortunately, Ericsity makes this pretty easy.

Step 1 is to plug in the controller. After a few seconds, you will see a new network available on your phone called WLED-AP. Join that network. From your browser navigate to 4.3.2.1 and select "WIFI SETTINGS".

The goal is to add this controller to your wireless network. Do this by adding your network name and password into the first two text boxes. I also updated the mDNS setting, but that's optional. Then save and connect.

Finally, install the WLED mobile application if you are going to manage this via your phone. On an android device you can find it in the Google Play store with the name WLED. Reconnect to your wireless network and the one you just added the controller to. Once installed and reconnected to the network, open the application.

If you have already installed and configured a controller, you'll see your previous controllers listed. To add the new controller, click on the + sign in the upper right. Then "Start Discovery". When the controller is found, you can press the check button.

The new controller will be listed on the main WLED page with the default name of WLED-SoundReactive. Select that controller by tapping it. Then select "Config" along the top and "Security & Updates" at the bottom.

Scroll all the way down to the bottom of this page and you should see that you have the following version installed:

WLED SR version 0.13.3

Update to 0.14.x¶

From the Security & Updates page, it makes sense to try and perform and update. If you click "Manual OTA Update" you'll notice that only version 0.13.3 is available via this link though. It's time to do a manual update!

I am utilizing the MoonModules branch because the README for the default install says that changes should be made against this branch.

Pull Requests should be created against the MoonModules mdev branch.

I performed this OTA upgrade from my computer by navigating to the mDNS I set up previously. It can also be accessed by the IP address of the controller in your browser.

At the time of this post, the current release was 0.14.1-b30. This was released approximately 6 months ago. I briefly skimmed through recent issues and pull requests to see if I should find a more recent build. There was an crash issue reported with the ripple effects. Since I like that particular effect, I decided to go with a newer build than the official release.

Official build¶

If you are going to stick with an official release, navigate to the release page and download the binary file you want to install. For the Ericsity, I found that the generic ESP32 build worked. If you want to use this, look for the file titled WLEDMM_0.14.1-b30.36_esp32_4MB_M.bin. You could also use the WLEDMM_0.14.1-b30.36_esp32_4MB_S.bin. Download this file.

Selecting a more recent build¶

Since I wanted a fix for the item I found, I opted to download a recent build of the mdev branch. This can be accomplished by navigating to the build pipeline and filtering for the mdev branch. This is on Github and available via this direct link to the mdev pipeline.

From here, select the build you want and scroll all the way to the bottom of the page. Find the binary you want to download. In my case, I wanted the firmware-esp32_4MB_M and download it. Once downloaded, extract it so that you can upload the .bin file within the WLED page.

Press the button!¶

In WLED's Security and Update page, select this binary and then click "Update!"

The update took less than a minute for me. While the new image was being installed, I was told not to close or refresh the page.

When the update is complete, the controller will reboot.

To confirm that everything has been updated, click on "Config". Immediately, you'll notice a lot more options. Scroll to the bottom and select "Security & Updates" and at the bottom you'll have an About section that lists the version you selected to install. Mine looks like this:

WLEDMM version 0.14.1-b31.38 ☾

Fixing Sound Reactive¶

During testing, I noticed that the sound reaction wasn't working. I found the AudioReactive plugin was disabled by default. This is an easy fix. In the WLED application (or web page), click on "Info" then click the power button icon next to "AudioReactive". However, this isn't enough to solve the problem. Click on "Config" then scroll down to AudioReactive. First make sure it is enabled (it should be after the power button selection above).

The digitalmic section needs to be modified. The Ericsity controller has the microphone on the following pins:

Microphone Type: Generic I2S
Pin I2S SD: 26
Pin I2S WS: 5
Pin I2S SCK: 21

I found that pin 5 was selected for another plugin. If you find this too, navigate back to Config page and select "Rotary-Encoder". I don't utilize this plugin, but it is part of the M build I downloaded. Disable the plugin and change the CLK Pin to be undefined. Then save this change and go back to the AudioReactive plugin and set it up with the pin layout above.

Finally, reboot the controller by selecting "Info" and rebooting the controller at the bottom.

Good to go¶

Now that the controller has rebooted, audio reactions are working again. The branch I've updated by controller to is actively maintained for the 0.14 branch, even if it is several months behind the main line. I'm satisfied with this for now and will begin experimenting with the led mappings that I need that started this entire upgrade process.

Python Gotcha: strip, lstrip, rstrip can remove more than expected

2024-03-29T09:00:00-05:00

Introduction¶

As a software engineer, you've cleaned your fair share of dirty strings. Removing leading or trailing spaces is probably one of the most common things done to user input.

In Python, this is done with the .strip(), .lstrip() or .rstrip() functions and generally looks like this:

>>> "     Andrew Wegner     ".lower().strip()
'andrew wegner'
>>> "     Andrew Wegner     ".lower().lstrip()
'andrew wegner     '
>>> "     Andrew Wegner     ".lower().rstrip()
'     andrew wegner'

That's pretty straightforward and nothing unexpected in going on.

Gotcha¶

The Gotcha is that each of these functions take a list of characters that can be removed.

>>> "Andrew Wegner".lower().rstrip(" wegner")
'and'

What happened? Why wasn't the result just

'andrew'

Explanation¶

Read line from the documentation again, carefully:

A list of characters

Not a list of strings.

This is explicitly spelled out in the documentation, with an example, showing what the implications are. However, for a new developer, it's unexpected behavior. After all, these seem like intutive functions.

The example with my does the following:

Receives a list of characters to remove. In this case it is all letters in my last name, plus the space character: wegner
Lower case all letters in the input string, resulting in andrew wegner
From the right hand side of the string, begin removing characters that are in the input list. Stop when you encounter a character not in the list. In this case that means that rengew wer are removed (right to left) and then the d in andrew is encountered so that rstrip function stops.
Return the remaining string of and

Solution¶

Python has two functions that will correctly remove a string - .removesuffix() and .removeprefix() for right and left side removals.

>>> "Andrew Wegner".lower().removesuffix(" wegner")
'andrew'

These two functions were introduced in Python 3.9 as part of PEP-616. In the PEP, it explicitly calls out the confusion users have about the *strip() functions and how they behave. These two were introduced to allow the desired behavior.

One important note is that these two remove* functions will only remove at most one instance of the string.

>>> "Andrew Wegner Wegner".lower().removesuffix(" wegner")
'andrew wegner'

Installing the Mobile Security Framework (MobSF) on Windows using Docker

2024-02-19T15:00:00-06:00

Introduction¶

I recently needed to poke around an APK. I'd done this sort of thing before, but it's been years so the process of setting up an Android environment was unfamilar. Instead of spending a day reimmersing myself for what was (hopefully) going to be a 10-15 minute thing, I turned to another tool: Mobile Security Framework (also known as MobSF)

Setting up Docker¶

There are countless blog posts, articles, and YouTube videos on how to set up the perfect docker environment. This was a quick task. I didn't need "perfect", I needed good enough. At the time, I was on a Windows 11 machine, so that's what I had to work with. The goal was to get Docker operational as quickly as possible.

Why? MobSF has a quick setup with two lines of Docker code.

Using the Install Docker for Windows documentation, I downloaded the installable and started the process. The installation took a couple minutes on the device I was on and required me to log out and back in. Total install time was less than 5 minutes.

Installing MobSF¶

Once logged back in, I fired up the command line. The MobSF README provides two lines to execute on the command line to pull the latest image. I didn't go through the process of setting up a dymanic analyzer. I didn't think I'd need it (and as luck would have it in this case, I was correct).

Run the two commands:

docker pull opensecurity/mobile-security-framework-mobsf:latest docker run -it --rm -p 8000:8000 opensecurity/mobile-security-framework-mobsf:latest

This will pull the latest version of MobSF and then run it on port 8000 on the local machine. Once it's running you can access it in your browser by going to http://127.0.0.1:8000

Find your APK¶

Theorectically, you'll be running this against an APK you've built and developed. In that case, you'll have the APK handy and can load it into the UI. I, unfortunately, didn't have that luxuary. Fortunately, getting the APK is relatively simple.

Go to the Google Playstore and search for the application
Navigate into the details page of the application (you should see the option to install it)
Copy the URL to this page
Go to APKCombo
Paste the URL you copied in step 3 instal the appropriate text box and click "Generate Download Link"
After a few seconds, click the download icon next to the link that was generated

You now have the APK to use.

Using MobSF¶

With the APK in hand, and MobSF running, navigate to http://127.0.0.1:8000. Upload the APK and wait a couple of minutes. If you are watching the logs, you'll see it performing multiple steps behind the scenes. Once it's done analyzing the APK, it'll drop you to the dashboard.

The dashboard has a lot of detail about the APK. Scroll through to see the permissions it's utilizing, certificates attached to it, potential security vulnerabilities, code analysis, detected URLs in the code, hard coded secrets, and strings. All of these would help a team build a better and more secure mobile application.

Conclusion¶

For my purpose, this 15 minute excersice not only answered the question I was seeking it also raised a few items that will need to be corrected before the next release. Introducing security practices with a tool like MobSF would be incredibly benefitial to a team tasked with building an Android application. Further, adding in the dynamic analysis capabilities can provide more details on areas to improve.

While I didn't need those to answer my question, I'd enourage someone looking to introduce this to a team, to look at the capabilities of MobSF

Python Gotcha: Modifying a list while iterating

2024-01-03T15:00:00-06:00

Introduction¶

A fairly common task an engineer may need to do is to remove all numbers from a list below a threshold. It's a simple enough task that anyone could accomplish - iterate over the list, check the value, if it's below the threshold remove it. With Python you can do this in a for loop.

Gotcha¶

Let's check code for this simple problem.

numbers_list = [0, 9, 2, 1, 5, 1, 10, 4, 3]
for index, value in enumerate(numbers_list):
    print(index, value, value < 5, 'Deleted' if value < 5 else 'Kept')
    if value < 5:
        del(numbers_list[index])

print(numbers_list)

It'd be reasonable to expect an final list that looks like this:

[9, 5, 10]

But, the actual output is:

[9, 1, 5, 10, 3]

What is happening?

Explaination¶

The (unconsious?) assumption that the developer is making with this piece of code is that the final list must be in numbers_list. Since we're able to iterate and modify, there is no need for a temporary variable or a need to worry about list copy gotchas. But, let's go through this one iteration at a time.

Starting list: [0, 9, 2, 1, 5, 1, 10, 4, 3]

| Iteration | Index | Value | Value < 5? | Action | List after iteration      |
| --------- | ----- | ----- | ---------- | ------ | ------------------------- |
| 0         | 0     | 0     | True       | Delete | [9, 2, 1, 5, 1, 10, 4, 3] |
| 1         | 1     | 2     | True       | Delete | [9, 1, 5, 1, 10, 4, 3]    |
| 2         | 2     | 5     | False      | Keep   | [9, 1, 5, 1, 10, 4, 3]    |
| 3         | 3     | 1     | True       | Delete | [9, 1, 5, 10, 4, 3]       |
| 4         | 4     | 4     | True       | Delete | [9, 1, 5, 10, 3]          |

End of iteration as list has no more elements

Looking at this, it's a little clearer what's going on. Let's go through there one iteration at a time.

On the first iteration, the value of Index 0 is 0, so it's deleted. And the problem starts now.
On the next iteration, the value of Index 1 is 2 (not 9). In the previous iteration, Index 0 was deleted, which means everything has moved up one index. The value of 9 was never checked. Instead, we look at a value of 2. It's less than our threshold so it's also deleted.
On the third iteration, the value of Index 2 is 5. Again, we've skipped a value (1) and it's never even checked. This iteration doesn't result in a deletion because 5 is not less than 5.
On the forth iteration, the value of Index 3 is 1. This is deleted as expected.
On the fifth iteration, the value of Index 4 is 4. We've skipped another value (10) and this iteration's value is less than our threshold so it's deleted.
There is no further iteration, because the last one has made the final value the last index, so the final value is never checked.

We skipped checking several values because we messed with indexes that were either being operated on or hadn't been operated on yet. This results in unexpected behavior.

Solution¶

A quick solution is to add the values that exceed our threshold to another variable:

numbers_list = [0, 9, 2, 1, 5, 1, 10, 4, 3]
larger_list = []
for index, value in enumerate(numbers_list):
    if value >= 5:
        larger_list.append(value)

print(larger_list)

This results in the expected list of [9, 5, 10]

Another solution, assuming that the result needs to sit in a variable with the same name, is to use a list comprehension. This does the same thing as above, but keeps the code more compact. It also works because the final result is assigned to a new variable object with the same name.

numbers_list = [0, 9, 2, 1, 5, 1, 10, 4, 3]
numbers_list = [nums for nums in numbers_list if nums >= 5]
print(numbers_list)

This results in the expected list of [9, 5, 10]

Python Gotcha: List Copy Problems

2023-12-20T15:30:00-06:00

Introduction¶

This problem isn't limited to only lists. Any mutable object in Python can be impacted by this "gotcha". The problem is setting one mutable object equal to another doesn't make a copy. It assigns a new variable to the same object as the original. Let's take a look and see how this appears.

The First Gotcha¶

The first example is pretty simple. Let's make a copy of our list

>>> list1 = [1,2,3,4,5]
>>> list2 = list1
>>> list1
[1, 2, 3, 4, 5]
>>> list2
[1, 2, 3, 4, 5]

That looks like I have two lists, both with the same set of values. But, let's modify the second list and change the first element in the list to be 6

>>> list2
[6, 2, 3, 4, 5]

But what about list1? It also has changed.

>>> list1
[6, 2, 3, 4, 5]

This is occuring because, as mentioned, list2 is assigned to the same object as list1. Any change to either, changes the object that both are assigned to.

>>> print(f"list1 id: {id(list1)}")
list1 id: 1497534800448
>>> print(f"list2 id: {id(list2)}")
list2 id: 1497534800448

The First Solution¶

This can be solved a few different ways. The important take away, though, is that you have to create a new object with the original values. I can think of a few ways to handle this. This article doesn't consider performance for these basic lists.

One option is to utilize the list() method and pass it the initial values from list1

>>> list3 = list(list1)
>>> list3
[6, 2, 3, 4, 5]
>>> list3[0] = 0
>>> list3
[0, 2, 3, 4, 5]
>>> list1
[6, 2, 3, 4, 5]
>>> print(f"list1 id: {id(list1)}")
list1 id: 1497534800448
>>> print(f"list3 id: {id(list3)}")
list3 id: 1497531282368

A second option is to slice() the entire original list into a new variable.

>>> list4 = list1[:]
>>> list4
[6, 2, 3, 4, 5]
>>> list4[0] = 0
>>> list4
[0, 2, 3, 4, 5]
>>> list1
[6, 2, 3, 4, 5]
>>> print(f"list1 id: {id(list1)}")
list1 id: 1497534800448
>>> print(f"list4 id: {id(list4)}")
list4 id: 1497534798208

Lastly, the list object has a copy() method that you can utilize.

>>> list5 = list1.copy()
>>> list5[0] = 0
>>> list5
[0, 2, 3, 4, 5]
>>> list1
[6, 2, 3, 4, 5]
>>> print(f"list1 id: {id(list1)}")
list1 id: 1497534800448
>>> print(f"list5 id: {id(list5)}")
list5 id: 1497534799552

With each of these three options, you can see that we created new objects because each has a unique object id. Changing one of these copies does not change the original's values.

The Second Gotcha¶

You, a savvy developer, know all this though. You also have a more advanced data structure. You have a list of lists.

[
    ['first element', 1, 2, 3],
    ['second element', 4, 5, 6],
    ['third element', 7, 8, 9],
]

Since you know about this gotcha, you are going to make a new object and copy the values to the new object.

>>> deeplist1 = [
...     ['first element', 1, 2, 3],
...     ['second element', 4, 5, 6],
...     ['third element', 7, 8, 9],
... ]
>>> deeplist2 = deeplist1.copy()
>>> print(f"deeplist1 id: {id(deeplist1)}")
deeplist1 id: 1497534571008
>>> print(f"deeplist2 id: {id(deeplist2)}")
deeplist2 id: 1497534577536

Now, with two different objects, is it safe to modify an element in one of the sublists? Turns out, it is not.

>>> deeplist2[0][0] = "NOT 3RD"
>>> deeplist2
[['NOT 3RD', 1, 2, 3], ['second element', 4, 5, 6], ['third element', 7, 8, 9]]
>>> deeplist1
[['NOT 3RD', 1, 2, 3], ['second element', 4, 5, 6], ['third element', 7, 8, 9]]

But why? The ids of deeplist1 and deeplist2 are different. This is true, but the sublists are not. Each sublist is assigned the same object id as the original.

>>> id(deeplist1[0][0]), id(deeplist2[0][0])
(1497534796864, 1497534796864)

The Second Solution¶

Utilize the deepcopy() method to make a copy of your original object.

>>> import copy
>>> deeplist3 = copy.deepcopy(deeplist1)
>>> deeplist3[0][0] = "NEW VALUE"
>>> deeplist3
[['NEW VALUE', 1, 2, 3], ['second element', 4, 5, 6], ['third element', 7, 8, 9]]
>>> deeplist1
[['NOT 3RD', 1, 2, 3], ['second element', 4, 5, 6], ['third element', 7, 8, 9]]

Using deepcopy() we can safely make a copy of our list of lists and modify both the original and the copy without impacting the other.

Python Gotcha: Timezone Naive Functions like utcnow() and utcfromtimestamp()

2023-11-23T10:15:00-06:00

Introduction¶

There are two common functions in that are naive datetime objects. The behind these types of objects is that timezones don't matter. Perhaps you always assume that datetimes are in a set timezone. Or, you are aware of timezones and want to make things easy on yourself so always assume datetimes are set to UTC.

This is your mistake.

The utcnow() and utctimestamp() functions are naive, but other datetime functions are not.

The Gotcha¶

>>> from datetime import datetime
>>> dt = datetime.utcfromtimestamp(0)
>>> dt
datetime.datetime(1970, 1, 1, 0, 0)
>>> dt.timestamp()
21600.0

What happened here?

This was run on a computer in the Central Standard Time time zone. First, I created an object set to the UNIX epoch - Jan. 1, 1970. Since I utilized utcfromtimestamp() to do this, there is no timezone data attached to this object. When I use the standard timestamp() function, Python sees there is no timestamp to utilize, so it uses the system configured timezone. Hence the 21,600 second offset from the epoch.

This didn't always happen though. Sometime between Python 2 and Python 3 this behavior changed. This assumption that "no timezone means local timezone" introduced this gotcha. In my opinion isn't a rather dangerous one too.

Solution¶

Utilize explicit time stamps everywhere. From The Zen of Python - PEP 20:

Explicit is better than implicit.

>>> import datetime
>>> dt = datetime.datetime.fromtimestamp(0, datetime.UTC)
>>> dt
datetime.datetime(1970, 1, 1, 0, 0, tzinfo=datetime.timezone.utc)
>>> dt.timestamp()
0.0

Notice that by adding datetime.UTC to my fromtimestamp() call, I get a datetime object with tzinfo. This makes the conversion from the datetime object back into ta timestamp safe.

This is the way that Python is moving. In an effort to correct these unsafe assumptions that were introduced early in Python 3, the naive datetime functions have been deprecated. If you were to run my first block of code in Python 3.12, you receive a deprecation warning.

>>> dt = datetime.utcfromtimestamp(0)
<stdin>:1: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).

Notice that the message is telling you to add the timezone information and to utilize fromtimestamp(). In the coming year - years? - both the commonly seen utcnow() amd utcfromtimestamp() will be removed.

In my opinion, as someone that has fought this bug in the past, it's a welcome change. Timezone assumptions are difficult to troubleshoot. Assumptions about underlying behavior make it a challenge. Running the same script in different timezones, like you would with a geographicly diverse team, makes it even harder. It's especially annoying if one of your coworkers is utilizing UTC as their time zone so that the script behaves properly for them but no others.

I have code bases that will need to make this change, but in the end, they will be easier to maintain with the removal of the utc* functions.

Ruby Gotcha: Operator Precedence

2023-10-23T10:45:00-05:00

Introduction¶

In one of my previous roles, the team utilized Ruby. I was not an individual contributor at that point in my career, so my exposure to it was less than much of my team. However, during my tenure, I noticed several "Gotchas" of the Ruby language. Each language has these in their own way, but in this post I'm going to cover one specific one.

What's the difference between and and &&? What about the difference between or and ||?

The Gotcha¶

At first pass, these two blocks of code appear to do the same thing. Let's take a look.

irb(main):003:0> true && false
=> false
irb(main):004:0> true and false
=> false

By the rules of boolean logic, both and and && statements return false.

irb(main):005:0> true || false
=> true
irb(main):006:0> true or false
=> true

Boolean logic holds true for the or and || statements and return true.

But, what happens if we assign the result of this to a variable?

irb(main):007:0> result = true and false
=> false
irb(main):008:0> result1 = true && false
=> false
irb(main):009:0> result
=> true
irb(main):010:0> result1
=> false

Now we see the problem. With the and operator, result = true. With the && operator, result1 = false.

Looking at the or/|| operators quickly,

irb(main):011:0> result2 = true or false
=> true
irb(main):012:0> result3 = true || false
=> true
irb(main):013:0> result2
=> true
irb(main):014:0> result3
=> true

This all looks reasonable. But, let's swap the order of our true/false in the statement.

irb(main):015:0> result2 = false or true
=> true
irb(main):016:0> result3 = false || true
=> true
irb(main):017:0> result2
=> false
irb(main):018:0> result3
=> true

Here we see the differences again, with the or operator setting result2 = false and the || operator setting result3 = true

What is going on?¶

and and && / or and || are the same thing, but they have different orders of precedence. The (not well formatted in my opinion) documentation shows the precedence table for Ruby.

The key things to point out here:

&& and || are above and and or, meaning they will be evaluated first.
= is also above and and or (but below && and ||)

Let's look at one of the examples again:

irb(main):015:0> result2 = false or true
=> true
irb(main):017:0> result2
=> false

With the order of precedence in mind, this is what happens:

result2 is assigned the value of false because = occurs before or
The remainder of the statement is evaluated as true

This was hidden when true and false was swapped because result2 was assigned a value of true and then the remainder of the statement was also evaluated to true.

How do you prevent this?¶

There is very few reasons to utilize the English words and and or in Ruby's logical evaluations. Instead, these should be utilized as flow control modifiers (think if and unless) and not for boolean logic.

Here's an example of how it should be utilized for flow control

irb(main):019:0> numeral = 10 && numeral / 5
(irb):19:in `<main>': undefined method `/' for nil:NilClass (NoMethodError)
        from C:/Ruby32-x64/lib/ruby/gems/3.2.0/gems/irb-1.6.2/exe/irb:11:in `<top (required)>'
        from C:/Ruby32-x64/bin/irb:33:in `load'
        from C:/Ruby32-x64/bin/irb:33:in `<main>'
irb(main):020:0> numeral = 10 and numeral / 5
=> 2

In our first example, with &&, the statement is broken up like this

numeral = (10 && numeral) / 5

This won't work because numeral doesn't have a value when && is evaluated. Contrast this with the second version where the code:

Assigns a value of 10 to numeral
Divides numeral by 5

While basic, it should get the point across. Another example could be something like this:

user = User.find_by_email(email) and user.send_email!

Here, we are only sending an email to the user if we find their information. No information, no email is sent.

Python Gotcha: Comparisons

2023-10-18T10:15:00-05:00

Float Equality Comparisons¶

Like every other programming language, Python can't accurately represent floating-point numbers. I'm sure many Computer Science students have lost many hours learning how floating representation works. I remember that class well.

In any case, let's get into the problem with comparing float values in Python and how to handle it.

>>> 0.1 + 0.2 == 0.3
False

You, me, and anyone with a few years of elementary school under their belt can see that this should be True.

What's happening here? We can see the problem by breaking down the component parts of this comparison.

>>> 0.3
0.3
>>> 0.1 + 0.2
0.30000000000000004

And now we see the floating point representation that is causing a problem.

So, how do we deal with this?

Decimal¶

There are a couple options, both of which have their drawbacks. Let's start with Decimal.

The decimal module provides support for fast correctly rounded decimal floating point arithmetic.

This sounds good, but an important gotcha here too is how it handles numerals vs. strings.

>>> from decimal import Decimal
>>> Decimal(0.1)
Decimal('0.1000000000000000055511151231257827021181583404541015625')
>>> Decimal('0.1')
Decimal('0.1')

This means, to accomplish the comparison above, we need to wrap each of our numbers in a string.

>>> Decimal('0.1') + Decimal('0.2') == Decimal('0.3')
True

That's annoying, but does function.

isclose¶

Python 3.5 implemented PEP 485 to test for approximate equality. This is done in the isclose function.

>>> import math
>>> math.isclose(0.1+0.2,0.3)
True

That's cleaner than wrapping everything in strings. But, it also is more verbose than a simple == statement. It makes your code less clean, but does provide accurate comparisons.

is vs. ==¶

Another comparison that I've commonly seen misapplied is developers using is when they mean ==. Put simply, is should ONLY be used if you are checking if two references refer to the same object. == is used to compare value by calling underlying __eq__ methods.

Let's see this in action:

>>> a = [1, 2, 3]
>>> b = a
>>> c = [1, 2, 3]
>>> d = [3, 2, 1]
>>> a == b
True
>>> a == c
True
>>> a == d
False

So far, nothing unusual. a has the same values of b and c and different values from d. Now let's use is

>>> a is b
True
>>> b is a
True
>>> a is c
False
>>> a is d
False
>>> b is c
False

Here, the only True statements are the comparison between a and b. This is because b was initialized with the statement b = a. The other two variables were initialized as their own statements and values. Remember, is compares object references. If they are the same, it returned True.

>>> id(a), id(b), id(c), id(d)
(2267170738432, 2267170738432, 2267170545600, 2267170359040)

Since a and b are the same object, we get a True on their comparison. The others are different, hence the False.

nan == nan¶

nan, or Not a Number, is a floating point value that can not be converted to anything other than a float and is considered not equal to all other values. It's a common way to represent missing values in a data set.

There is a key phrase in that description above that is the basis for this Gotcha:

is considered not equal to all other values

It's common for software to check if two values are equal to one another prior to taking an action. For nan, that does not work:

>>> float('nan') == float('nan')
False

This prevents code like this from entering the if block of an if/else statement

>>> a = float('nan')
>>> b = float('nan')
>>> if a == b:
...   .. ## Do something if equal
... else:
...   .. ## Do something if not equal

In this example, they are never equal.

This leads to an interesting, if not unintutive, way of checking if a variable is a nan value. Since nan is not equal to all other values, it is not equal to itself.

>>> a != a
True

Like I said, "Interesting". But, when your code is looked at by others it's also "confusing". There is an easier way to show that you are checking for a nan value. isnan

>>> import math
>>> a = float('nan')
>>> b = 5
>>> c = float('infinity')
>>> math.isnan(a)
True
>>> math.isnan(b)
False
>>> math.isnan(c)
False

To me, that's a much clearer check that we want to see if the value is nan. It's likely you aren't just passing nan to a single variable. You're probably using a library like NumPy or Pandas. In that case, you have functions in each of those libraries that can check for nan in a performant way.

In NumPy the function has the same name but in the NumPy library: numpy.isnan(value).
In Pandas, the function has a slightly different name: pandas.isna(value)

Conclusion¶

Comparisons aren't always as straight forward as we'd like. I covered a few common comparison problems in Python here.

Floating point comparisons are common across languages. Python has a few ways of making this easier for developers. I recommend utilizing isclose as it keeps the code a bit cleaner and eliminates the need to wrap numbers in strings if using the Decimal module.

is should only being used to check if two items are referring to the same object. In any other case, it's not doing the check you want it to be doing.

Lastly, nan is equal to nothing else. It's important to be aware of that before you start comparing values in your dataset to one another.

Python Gotcha: Join vs Concat

2023-10-12T10:45:00-05:00

The Problem¶

Here's a somewhat contrivied example to demonstrate a problem. I need to create a string with the word word 100,000 times. What's the fastest way to generate this string? I could use string concatenation (simply + the strings to one another). I could join a list with 100,000 items.

My proposed code for this example is below

def concat_string(word: str, iterations: int = 100000) -> str:
    final_string = ""
    for i in range(iterations):
        final_string += word
    return final_string

def join_string(word: str, iterations: int = 100000) -> str:
    final_string = []
    for i in range(iterations):
        final_string.append(word)
    return "".join(final_string)

Results¶

In concat_string, I iterate 100,000 items with each iteration adding my word to the end of the string. In join_string, I append my word to a list on each iteration and then join the list into a string at the end.

Running each function through the built in profiler (cProfile) shows how these two functions perform.

>>> cProfile.run('concat_string("word ")')

      4 function calls in 1.026 seconds

Ordered by: standard name

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1    0.000    0.000    1.026    1.026 <string>:1(<module>)
     1    1.026    1.026    1.026    1.026 test.py:9(concat_string)
     1    0.000    0.000    1.026    1.026 {built-in method builtins.exec}
     1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

>>> cProfile.run('join_string("word ")')

      100005 function calls in 0.013 seconds

Ordered by: standard name

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1    0.000    0.000    0.013    0.013 <string>:1(<module>)
     1    0.009    0.009    0.013    0.013 test.py:16(join_string)
     1    0.000    0.000    0.013    0.013 {built-in method builtins.exec}
100000    0.004    0.000    0.004    0.000 {method 'append' of 'list' objects}
     1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
     1    0.000    0.000    0.000    0.000 {method 'join' of 'str' objects}

What's happening here?¶

join is more than 75x than the concatination method. Why?

String are immutable objects in Python. I talked about these in my last Gotcha Article about default parameters. This immutability means that a string can't be changed. concat_string does appear to be changing the string with each + action, but under the hood, Python has to create a new string object each iteration through the loop. That means there are 99,999 temporary string values - creating and discarding almost all of them immediately on the next iteration during the concatenation action.

join_string on the other hand, is appending 100,000 string objects to a list. But, only one list is created. The final join is only doing a single concatenation with all 100,000 strings.

What are the implications of this?¶

While this is a contrived example to show the problem, there are real performance impacts to string immutability that may not be obvious. There are other places where a new string is created commonly used in Python. A couple examples are f-strings, %s format specifiers and .format(). Each of these create a brand new string.

This doesn't mean you should avoid these, as the performance impact is only really obvious in situations where you are appending a lot of strings together. However, if you have a string formatting line in a loop, it's a potential area to focus on for performance improvements.

Python Gotcha: Mutable Default Optional Arguments

2023-10-06T10:15:00-05:00

The Problem¶

Without running this in your IDE, what does the following code output? Forgive the use of vins, I work in the logistics field and deal with vehicle identification numbers frequently.

def add_vin(this_vin: str, vins: list = []):
    if this_vin not in vins:
        vins.append(this_vin)
    return vins

add_vin("VIN1")
add_vin("VIN2")
print(add_vin("VIN3"))

You probably saw that vins is an optional argument and that it defaults to an empty list. Run this function three times without passing it a list to append to, and it's reasonable to assume that you have a final result of a list with a single item contained in it - VIN3

Unfortunately, that's wrong. The output is actually ["VIN1", "VIN2", "VIN3"]

Why?

What's going on?¶

In Python, default arguments are bound to a function at the time the function is defined. This is only done once. If you don't pass the optional argument in to the function call, the same list is changed when the function is called. This means that in the example above, a new list is not created each time the function is called without the optional vins argument.

We can see this by slightly modifying the code above to show the object ID we are operating on.

def add_vin(this_vin, vins=[]):
    if this_vin not in vins:
        print(f"id={id(vins)}")    # Print the object ID we are appending to
        vins.append(this_vin)
    return vins

Which outputs the following as it adds each VIN.

id=2954990867712
id=2954990867712
id=2954990867712
['VIN1', 'VIN2', 'VIN3']

Each time add_vin was called without passing it a list to operate on, the function will operate on the same default list.

Another example¶

The example above is basic, but gets the point across. However, let's look at something a bit more common. Connecting to a database with a connection object at the parameter.

def open_database(connection = make_connection(host='example.com')):
    # do database things with `connection`
    connection.close()

Well done. You've closed your connection. Your DBA will thank you. However, what happens next time you call open_database() in your code? After all, your connection information is there by default, you don't need to pass a new connection, right?

Again, wrong. You've closed the connection and it's the default argument. The next call to open_database will utilize the same connection object. A connection that is closed. Your database call will fail.

Last Example¶

One last example:

import datetime, time

def print_datetime(dt = datetime.datetime.now()):
    return str(dt)

print(print_datetime())
time.sleep(10)
print(print_datetime())

This is going to print the current datetime, sleep for 10 seconds, and print the new datetime - right?!

No.

2023-10-06 08:45:21.392973
2023-10-06 08:45:21.392973

Why?!

Again, an optional default argument is only bound once. Since a new value isn't passed to the function on the second call, the exact same value is used the second time through. Whoops.

What can I do about this?¶

The best option, in my opinion, is to set the default values to None. This is going to make it very obvious that there is a problem. In the first example, you'll get an exception because you can't append to None. In the second, your database calls will error because None doesn't contain connection information. In the third, you'll print None instead of an expected datatime.

You could pass an immutable object like a tuple or a frozenset, but even these have caveats to be aware of. For example, a tuple can contain something mutable in it (ie. a list). Personally, I prefer to utilize None.

Syncing Obsidian Notes across devices using Git and GitLab

2023-09-26T13:30:00-05:00

Introduction¶

I take a lot of notes. About five years ago I switched from utilizing notebooks, because I was going through them so quickly, to a Rocket Book. This feels very similar to a notebook, with the benefit that it is reusable. It also has an application that allows me to quickly scan a page and save it to one of several locations. It's very handy. The downside is that the OCR isn't perfect (oooor, my handwriting is bad) and the annoyance of cleaning the pages every week or so. I've found that if I don't let the pages dry completely, then the next time I go to use that page it has a fine film over it that makes writing a little more difficult. Think writing with a pen that is low on ink.

I discovered Obsidian a while ago. I've put a little bit of effort into utilizing it, but more recently I've committed to utilizing it more. The idea of all digial notes is enticing. Over time I'll see if it's helpful.

But, like anyone with multiple devices, I want my Obsidian vault on all devices. What if I have a thought while I'm away from my main computer? One option is to utilize the Obsidian Sync feature. It's got end to end encryption, version history and a monthly subscription fee. I can build something using my existing GitLab installation without that monthly subscription fee. Plus, since I'm the only user of this, I'm ok with a little trade off that a full service provides.

Repository Setup¶

GitLab¶

Within GitLab set up a new project by clicking on "New Project" in the upper right.

Select the option to create a new blank project.

Now we need to set up the project details. Select a name that is meaningful for you. I've utilized andy-notes in the screenshot below. I also created this without a README, because I'll generate the initial repository in a moment and push it up to the server.

Repository Initialization¶

This set of steps only has to occur once. All devices will be able to start with the next section.

Open the command prompt or terminal (depending on what your operating system calls the system console). Navigate to the location you want your notes to be stored. In my case, I created a new directory to match the name of my repository. I then created a .gitignore with the following content in the directory.

.obsidian/*
!.obsidian/app.json
!.obsidian/appearance.json
!.obsidian/config
!.obsidian/community-plugins.json
!.obsidian/core-plugins.json
!.obsidian/graph.json
!.obsidian/hotkeys.json

This should ignore the majority of the .obsidian hidden directory between my devices, but keep a few key items. I may update this over time, but for now this is working well.

Now, initialize the repository.

cd andy-notes
git init --initial-branch=main
git remote add origin git@<my.server.url>:andy/andy-notes.git
git add .
git commit -m "Initial Commit"
git push --set-upstream origin main

We're ready to start utilizing Obsidian and GitLab!

Obsidian Setup¶

First Open¶

Now that the backend repository is set up and it's been initially configured, all instances of Obsidian can utilize these sets of steps to start working.

If this is not the device you initilized the repository on, you need to perform an initial pull of the repository to this device so that it is a local folder.

git clone git@<my.server.url>:andy/andy-notes.git

Open Obsidian. Since this is the first time you are opening it, you'll be asked to select where your vault is stored. You want to select the "Open folder as vault" option.

Select the location that your repository has been cloned into.

Enable Plugins¶

Obsidian has support for community built plugins. There are plugins to do all kinds of things to improve your writing and note taking. I encourage the reader to look through or search for plugins that may help you. We, however, need one specific one to accomplish our goal of syncing the vault over to GitLab.

Obsidian Git

To install community plugins we need to enable this feature. In the lower left of the workspace, click the gears icon. Then click on "Community plugins", and finally "Turn on community plugins"

With this enabled, you can now browse the community plugins. To do so, click the browse button.

Search for Obsidian Git. While writing this, it was the first result. You should be looking for the one by Vinzent. Click on the card to pull up plugin details and potential actions.

Then click "Install". The installation process should only take a moment or two.

You'll receive a confirmation in the upper right if the install is successful.

Finally, enable the new plugin. The previous "install" button has changed to be "Enable". Click that.

Configure Obsidian Git¶

With the Obsidian Git plugin installed, it's time to configure it. There are a few quick settings to fill out. First, you need to click the "Options" button.

The first setting to change is how often you want the backup to GitLab to occur. This automatic back up is disabled by default. I set mine to be every 5 minutes.

Next, enable Pull updates on startup so that the instance of Obsidian that is running is always up to date. Your notes from other devices will be synchronized when Obsidian launches and you can get right back to writing.

Be sure to add the Author name and email address. This is the information that will be sent on each commit.

I also adjusted the autocommit message to include the hostname of the device performing the commit. I have been going back and forth on whether this is important enough for me to keep since I'm the only one that utilizes this vault. But, if you want to adjust the messages there are two instances where the commit message can be modified.

Results¶

After performing the steps above, this instance of Obsidian is ready to utilize GitLab to sync your vault for free. Each new instance only needs to have the Obsidian Git plugin installed, enabled and configured.

After you've configured your instance, when you open Obsidian the plugin will pull changes from the repository and let you know that everything has been updated.

Then on a scheduled basis - 5 minutes in my case - it will commit and push changes to GitLab.

In GitLab, this commit will should show up immediately.

Conclusion¶

I now have Obsidian set up on my main devices. I've enjoyed the ability to have my notes available as soon as I open the application. Overall, this was pretty easy to set up. It seems like a lot of steps, but the one time setup makes it much longer than installing and configuring a plugin for each new instance actually takes.

I'm trying to keep my notes digital and I'm hopeful that this new set up will help. I can move between computers for personal projects instead of wondering what I had found in my research because I left it on another tab on another machine and forgot to put it with the rest of my notes. Now, it just happens.

How to disable Grafana in Gitlab 16.3 Omnibus

2023-08-23T22:15:00-05:00

Introduction¶

I continue to love my self hosted instance of GitLab. I look forward to the 22nd of every month to pull down the newest dot update and enjoy the new features. As with any piece of software, things get deprecated and eventually disabled over time.

In 16.3 - released in August 2023 - the version of Grafana bundled with the self hosted version of GitLab reached the end of it's depreciation window and was disabled. GitLab added a check to prevent an update if it was still enabled.

Symptom¶

When performing an apt update, GitLab 16.3 will error out if Grafana is still enabled on your instance. The error looks like this:

Preparing to unpack .../gitlab-ee_16.3.0-ee.0_amd64.deb ...
* grafana[enable] has been deprecated since 16.0 and was removed in 16.3. The bundled Grafana is deprecated and no longer available. We recommond following
https://docs.gitlab.com/ee/administration/monitoring/performance/grafana_configuration.html#deprecation.
Deprecations found. Please correct them and try again.

The Problem¶

The documentation linked provides four steps to follow

To switch away from bundled Grafana to a newer version of Grafana from Grafana Labs:

Set up a version of Grafana from Grafana Labs.

Export the existing dashboards from bundled Grafana.

Import the existing dashboards in the new Grafana instance.

Configure GitLab to use the new Grafana instance.

I've removed the links in steps 2 and 3, because they really don't matter for this write up. Step 1 does not have a link, so it's left as and excercise to the user to install Grafana in their environment. Step 4, links to another article that provides a few more steps to follow.

On the left sidebar, expand the top-most chevron.

Select Admin Area.

On the left sidebar, select Settings > Metrics and profiling and expand Metrics - Grafana.

Select the Add a link to Grafana checkbox.

Configure the Grafana URL. Enter the full URL of the Grafana instance.

Select Save changes.

And that's it. That's the set of instructions provided by GitLab. Follow those, rerun your apt update and you'll run into the same symptom.

Solution¶

The missing step is hinted at in the error message itself: grafana[enable] has been deprecated

This is a setting in the GitLab configuration file. You will need to edit the config file and set the value to false.

Edit /etc/gitlab/gitlab.rb (or appropriate path)

Edit the grafana[enable] value to be false. On my config this was on line 1689, but I recommend you search for the string grafana to find it.

grafana['enable'] = false

Reconfigure GitLab by running gitlab-ctl reconfigure. Let this run and it should end with gitlab Reconfigured!

You can now resume the apt update that failed due to having Grafana enabled.

Once done, you will get the standard GitLab upgrade complete message and you'll be good to go.

Reading unsupported config value grafana.¶

Update: July 2024, GitLab 17.1

In the GitLab 17.1 update from June 2024, Grafana has been completely removed. If you still have grafana keys in the config file upon updating to 17.1, the upgrade will fail with

Mixlib::config::unknownconfigoptionerror: Reading unsupported config value grafana.

The solution to this is to edit /etc/gitlab/gitlab.rb and comment out (or remove entirely) all lines that start with grafana. Once this is done, save the file and attempt the upgrade again. Like when it was originally deprecated, this step will end with gitlab Reconfigured.

CoderByte assessments fall to ChatGPT

2023-02-04T23:30:00-06:00

Introduction¶

Today I'll be evaluating CoderByte and their publicly posted challenges against ChatGPT. I selected an easy, a medium and the single hard question from their list of open challenges and ran those through ChatGPT.

In previous articles in this series, I've shown how ChatGPT can easily solve interview assessments from LeetCode, TestGorilla, CodeSignal, Codility and HackerRank[hackerrankg]. I've said it in those articles, but I'll say it again: Companies need to evaluate their engineering candidates - all candidates, actually - apprpriately when they are hiring. Your engineers are using many tools to do their jobs and AI Tooling is one of the newest tools on the market. Removing a tool from your engineers is going to make them less efficient, and not testing your candidates on how they use these tools is going to mean you hire someone that had not fully demonstrated their capabilities.

ChatGPT should be treated as a tool, but it's also helpful to think of it as a junior level engineer. You need to check it's work. ChatGPT is known to be inaccurate, which is why it remains banned on Stack Overflow. There are a few instances where I had to do that below.

First a disclaimer: I was Director of Engineering at Woven Teams in 2022. Prior to that, I was a customer of Woven for over two years. I do not remember CoderByte being mentioned during my tenure and CoderByte does not seem to consider Woven a competitor, as Woven isn't listed as an alternative on their pricing FAQ page.

Easy - React Button Toggle¶

While I've tried to be mostly random in my selection of assessments during these tests, I intentionally selected this react assessment because I don't know react. I've liked these types of assessment questions in the previous articles because I means I can't apply my experience to "nudge" ChatGPT toward a solution.

The challenge here is to toggle a button between "On" and "Off". You're presented with a very small amount of boiler plate code.

I simply copy/pasted the prompt and the template into ChatGPT. It sent back this:

import React, { useState } from 'react';
import ReactDOM from 'react-dom';

function Toggle() {
const [toggle, setToggle] = useState(true);

function handleClick() {
    setToggle(!toggle);
}

return (
    <button onClick={handleClick}>{toggle ? 'ON' : 'OFF'}</button>
);
}

ReactDOM.render(
<Toggle />,
document.getElementById('root')
);

I copied that into the IDE provided by CoderByte and submitted the code.

This test took two minutes to complete. The bulk of that time was spot checking that ChatGPT didn't mess with the boiler plate code to badly.

The interesting this about the results is that even with the 10/10, the system says it only scored higher than 12% of other users. Does this mean that only 12% of the users passing this assessment are getting lower than a 10/10? To me, that's the sign of a bad assessment.

Medium - SQL Member Count¶

The medium challenge I selected was an SQL question. SQL is interesting, in my experience, in that the results returned on an assessment are a lot more uniform compared to a non-SQL coding assessment. This one wasn't a difficult one with joins.

The goal of this assessment was to return a list of names in alphabetical order, a count of the number of people that report to them, and the average age of their team members. I pasted the prompt, the expected table layout for the output and the table layout being selected from.

The first response was

SELECT ReportsTo, COUNT(ID) AS Members, AVG(Age) AS Average_Age
FROM maintable_YTTQH
WHERE ReportsTo IS NOT NULL
GROUP BY ReportsTo
ORDER BY ReportsTo;

This is where experience, and checking the results are important. The expected output showed that the average age should be an integer. It also showed that the average age column should be named Average Age not Average_Age (notice the lack of the underscore). I provided this guidance to ChatGPT

The AVG(Age) should be an integer not a float and The "Average_Age" field should be "Average Age" without an underscore.

It corrected both problems with a new query which I submitted.

SELECT ReportsTo, COUNT(ID) AS Members, ROUND(AVG(Age)) AS "Average Age"
FROM maintable_YTTQH
WHERE ReportsTo IS NOT NULL
GROUP BY ReportsTo
ORDER BY ReportsTo;

ChatGPT scored another 10/10.

Even with that extra round trip with ChatGPT, it took less than a minute to complete this assessment.

This assessment seems to have a slightly better split of user scores, which is progress I suppose.

Hard - Bracket Combinations¶

There is only one hard assessment available publicly. The goal for this assessment is to return the number of combinations you can make with a number of pairs of parenthesis. The example provided is:

The possible combinations of 3 pairs of parenthesis, namely: ()()(), are ()()(), ()(()), (())(), ((())), and (()())

I copied the prompt and boiler plate code into ChatGPT and it returned the following response:

def BracketCombinations(num):

    from math import factorial
    return factorial(2 * num) // (factorial(num) * factorial(num + 1))

# keep this function call here 
print(BracketCombinations(input()))

Experience here says that is odd. Technically speaking, Python will allow you to have that import statemnet inside a function, but if I saw one of my developers do it, I'd mention it during a code review. I was curious though and submitted it as is.

I also went back and told ChatGPT:

Write the function without importing factorial

It replied with this function:

def BracketCombinations(num):
    def fact(n):
        result = 1
        for i in range(1, n + 1):
            result *= i
        return result

    return fact(2 * num) // (fact(num) * fact(num + 1))

# keep this function call here 
print(BracketCombinations(int(input())))

I submitted this response too.

I only have one set of images for this challenge, because they both output exactly the same responses. Both solutions scored the same. Both solutions were better than 35.1% of other users. Both solutions had an O(n) complexity.

Showing the complexity is a nice touch. I will admit that I was expecting the solution that imported factorial to be rejected because it's using a built in function instead of rolling your own. Good for CoderByte for allowing a developer to use the built in libraries as an engineer would do in a production environment, instead of adding a false constraint.

Copy pasting to CoderByte kept this close to a minute.

Final Thoughts¶

Another code only assessment tool has been shown to be ineffective. The one thing this one does, that others does show publicly, is the time it took to solve a problem. I'm very surprised that a majority of these were solved in under ten minutes. Even so, solving a problem in less than a minute will probably raise a flag of some kind for a hiring manager giving this type of assessment. I guarentee that time is visbile to them.

These three assessments show how well ChatGPT can be in being your virtual junior engineer. With the react question, I have no idea if it's the more efficient way to solve the problem but it works. With the SQL question, experience identified two small problems and allowed me to provide feedback and get a corrected query in seconds. The third question turned in two different ways of solving the problem. Apparently, according to CoderByte, bother of which perform exactly the same.

The total time it took me to run these three tests was under five minutes. That's a great tool to have in my pocket.