Modernize Embedded Systems Engineering with DevOps [Small Business Edition]

Hi. My name is Jakub Zawadzki. I'm from Poland.

I'm an engineer with a passion for Embedded Systems & IoT projects, where hardware products play an important role. I have almost two decades of experience in these types of projects as a SW developer, HW designer and in the DevOps field.

If you are an Embedded Systems Engineer working solo or in a small team and want to learn more about how DevOps can help you in your daily life, then you are in the right place. I'm starting an educational project that might interest you.

What is the idea and where did it come from?

My long experience as a SW developer (and short experience as a HW designer) was always in small-scale projects done by small teams in small-scale businesses. I have experienced first-hand the successes of projects that went smoothly, as well as the failures of those that were inadequately managed and got out of hand.

For the last four years, however, I've been developing my skills in the DevOps field as part of a team within a large multinational corporation environment. I'm witnessing the ways in which DevOps can improve the development process for large-scale embedded systems used in the infrastructure of modern mobile networks, as well as simplifying the daily lives of embedded developers and project managers.

And I remember very well the problems I faced as an SW developer on smaller projects, and I see that most of them were very similar, just on a different scale. I believe these problems can be solved using the same DevOps techniques, but they need to be scaled down to fit small teams with fewer resources.

This is where the idea of pocket DevOps comes to life.

DevOps for small businesses and small teams
- where a small team starts from one person - You.
DevOps focused on Embedded Systems & IoT projects
- however, other project types could benefit too.
DevOps so small that you can hide it in your pocket (assuming that you have big pockets ;P)
- You don't need fancy servers or cloud infrastructure. All you need to get started is your computer. You can expand later if you need to.
- You don't need a separate department of DevOps Engineers to maintain your setup. You can do it yourself as part of your daily tasks - it's not rocket science, and it won't take you long to set up if you know what you're doing.

What can you expect to find in this project?

I won't pretend that I have all the answers. I don't have the full picture yet. I just have a few ideas about where to start. But I want to embark on this journey, and I want you to join me. I want to combine my current and previous experience, learn through brainstorming and experimentation, and share with you what I've learned so far.

Most of my time I spend researching and curating valuable DevOps-related resources from across the internet. I share all of these findings through my newsletter, along with my own insights and commentary. This way, you get both my original content and carefully selected resources from other experts in the field, all delivered directly to your inbox.

It is my side project that I work on in my free time, after my day job. I plan to create my own content as frequently as possible, but don't expect that I will publish it regularly. I believe in quality over quantity, so I prioritize delivering valuable, well-thought-out content over maintaining a strict publishing schedule.

The newsletter is completely free, and it's the best way to stay up to date with all my publications. Instead of checking multiple platforms regularly, you'll receive a notification whenever I publish something new or find something worth sharing. This way, you won't miss any valuable content.

I promise that I won't send you any spam. You can unsubscribe at any time.

What are the elements of this project?

Newsletter - This is the base of the project and the main communication channel. If you want to stay up to date with all my publications or keep in touch, this is the best place to do so.
Blog - You will find here all the archived issues of the newsletter, as well as my public notebook where I post additional information that may interest you.
Youtube - My video channel that is an extension of the materials available on the blog. Following the principle that a picture is worth a thousand words, it is sometimes easier to show something than to try to explain it in words.
GitHub - I post random snippets of source code here. They are either linked to blog articles or YouTube videos, or they are small, standalone projects by themselves.
LinkedIn - My professional profile. In case you're interested in learning more about me, especially what I do besides create content for this project.

But what exactly is DevOps? And why should you care?

DevOps

DevOps is a mindset, a culture, and a set of technical practices that will help your team maintain a high quality of product development. Use the following practices and you will not regret it. They will take your development process to a higher level.

Agile Project Management

As a risk mitigation strategy.
Iterate, monitor and adapt - spend time to constantly improve the way you work.

Prototype Driven Development

Build a prototype first - minimize the risk of project failure.
Verify the idea before getting started with a full-scale feature or project.

Behavior Driven Development

Refine and verify your requirements.
Make sure that you are building the right product.

Test Driven Development

Build your product with testability in mind right from the start.
Build trust in your code - give yourself the confidence to modify the code without worrying about breaking it.

Centralized Repositories

Single source of truth for your project files.
Ensure consistent releases - always build and test from the same centralized project configuration.

Managed Environments

Ensure that each member of the development team is using the same tools and environment configurations.
Use the same configuration across your CI/CD pipelines for consistent builds and releases.

Code Review

Clean code = higher product quality - let someone else look at your code before you deliver it.
Use automated tools for code analysis - a computer will analyze your code quicker and better than any human developer.

Live Documentation

Document everything you do. In a few months you won't remember what you did today. Write it down for your future self.
Leave notes in your design files and later, use automated tools to transform it into living documentation.

Automation

Your time is precious - if you have a cyclic task that is difficult or tedious to perform, automate it.
Focus on important tasks that will drive you forward, and let automation do the rest for you.

CI/CD Pipelines

Be sure that your product is always in releasable state.
Or be aware why it isn't.

Over-The-Air Updates

Make sure your product is bug free.
Even after you have shipped it.

Monitoring

Implement feedback loops.
Make sure that your project planning is based on real data, not on assumptions.

How is DevOps for Embedded Systems & IoT different from regular DevOps?

All of the above points will apply to these types of projects. The real difference is that you are building not only the software (SW) part but also the hardware (HW) and firmware (FW). These three must interact closely together. Therefore, there is a need for additional practices in your product development process.

Interdisciplinary Teams

Gather HW, FW and SW engineers in one place and let them build together iteratively, one increment of SW, FW and HW in parallel.

Testing Firmware on Host

Build your FW architecture in this way, so you can test it on the development machine and in CI/CD pipelines without having to use real HW.

Hardware-in-the-Loop Testing

Use simulated peripherals to automate FW integration testing on real HW.

JTAG Boundary Scan

Make it possible to test HW prototypes when the FW is not ready yet.

Fault Injection

Add the ability to simulate conditions that go beyond the capabilities of the real-world components.

System Level Simulation

You can have as many virtual products as you wish, on demand, ready to test your business logic.

Digital Twin

Build a digital copy of your product. You will be able to test it on extreme cases, without the risk of damaging it or injuring anyone.

Factory Tests

Before sending a product to a customer, test every single piece.

Failure Prediction

Collect logs from your devices in one place and analyze them looking for anomalies.

Resources Monitoring

Monitor the resource usage of your device (CPU, RAM, Flash, ...) on every commit to the master. If you need to optimize it later, you will know which parts of code to look at first.

Managing OTA Firmware Updates

Gain full control over whether, when and to which FW version your product will be updated. And verify that it has been done without errors.

Firmware Self-Test

Check that the FW update was successful and does not cause any problems. If not, roll back to the last known working state.

A little more about me and my experience.

Jakub

I consider myself more of an engineer than just a software developer. When working on a project, if possible, I like to have a broader perspective on every aspect of system design. I have knowledge in many subjects, which gives me the ability to work in different roles and on different types of projects.

I have more than 15 years of professional experience in software development in Python, C, C++, Delphi, and more, including over 2 years in bare-metal programming of embedded systems based on ARM, PIC, and Renesas RL78 family microcontrollers.

Over 4 years (and counting) of experience as a DevOps engineer, working to support software teams and make sure products are delivered successfully by implementing tools, frameworks, and technologies to enhance code quality and developer productivity.

2 years of experience in technical leadership, team coordination, and working with the Agile methodology as a Scrum Master.

I also have experience in designing digital electronics hardware (including 2 years in commercial projects). I don't have plans to work as an electronics designer anymore, but that knowledge has proven to be useful many times in work on embedded projects.

I'm a big Linux enthusiast on desktop, servers, and embedded systems. I use it in my everyday work.

I am a Polish native speaker with English as my second language. Bear this in mind if you notice any mistakes in my publications.