Diary of a Data Engineer

In 2011, I published the Diary of a DBA. Back then, the drama revolved around LTO backup tapes, SQL Server 2000 databases, rigid code, and a boss who wouldn't set foot in the office before 11:32 a.m. Fifteen years have gone by. The DBA evolved into a data engineer, the old magnetic tape became a lakehouse, MSN gave way to Slack… and me? I keep fooling myself that, this time, I'll leave on time. Any resemblance to reality is still not a mere coincidence.

07:50. I open the laptop ten minutes early, in the sweet illusion of getting ahead before the daily standup. Slack greets me with 137 notifications. A coworker carpet-bombed the #random channel with fourteen soccer videos before eight; now he sends me a "good morning!" stitched onto a master's thesis about yesterday's VAR blunders. Some things run perfectly on any infrastructure.

08:10. Quick triage: three @channel mentions that were anything but urgent, eight messages from a user swearing the dashboard "showed a different number yesterday," HR's automatic happy-birthday for someone I've never met, and two recruiters on LinkedIn offering me a DBA position (the algorithm never forgets my past). The overnight system alerts resolved themselves. That's great, but deeply suspicious.

08:50. The processing lag in our message queue system passed 40 million events. The reason? The client's DBA restarted the branch-office server without warning — fifteen years later, that sentence hasn't changed a single comma. I restart our connection, the pressure eases, and our two billion daily events flow in peace again. Meanwhile, in the support channel, I watch the veterans teaching the intern to tell the client that "the system is unstable due to a mishap in the user's operation" (translation: he clicked the wrong button).

09:05. Time to be the company's official debt collector. There's code sitting in review for a week, a cloud capacity increase request that's overdue, a system upgrade postponed for three sprints (I've lost count), and a firewall rule blocked "by mistake" for the third time. Oh, and of course: my ticket for a second monitor, opened with HR three weeks ago. Last time, they sent me a mousepad.

09:25. Coffee break. The pod ran out; I reheat yesterday's, honoring the sacred traditions of IT. From the video call next to me, I hear an analyst drop the classic on the client: "Do you want me to fix it the right way or the fast way?" I think to myself: here comes trouble.

09:50. Urgent ticket! Someone at the client ran an unfiltered update and twelve thousand records "vanished." In 2011, that meant hunting down a backup tape, borrowing a server, and losing the entire afternoon. Today, thanks to modern data time travel technology, I fix it with a 30-second command. That is… I would fix it, if the cleanup job hadn't wiped the history at six in the morning. The cloud backup notifies me: "recovery started, available within 12 hours." The worst part? The analyst promised the client he'd fix "the thing" in five minutes. The legacy server where this disaster lives goes by the affectionate name Lady Aracnea II.

10:15. I lose a bet. Every day we bet on what time the coworker known as "Bro" joins the daily. Today: 10:15, camera off, claiming "connection problems." Far from his record — 11:40, with a frozen profile photo pretending to be video. The intern took my seventy-five cents over PIX.

10:45. The twelve thousand records came back: a lost copy survived in a forgotten corner of the system. Thank you, past me. I go back to what I was doing: refactoring a monstrous query that hits a table with eight billion rows all at once, freezing everything. The columns are named tmp_final_v2 and saldo_NOVO_usar_esse. Bad habits have evolved, but they haven't died.

11:15. Thermostat War, hybrid edition. In the physical office, half want the AC at 18°C, the other half are wearing coats. Remotely, the war is "camera on × camera off" in the meeting with the client. I hear the sound of swords being sharpened in the #general channel.

11:32. The partner's dot turns green on Slack. Fourteen "typing…" icons vanish at once. In the background, you can almost hear the Imperial March.

11:45. Debate in the #lunch channel: iFood with a coupon vs. home-packed meal vs. Japanese by the kilo. I say I'm up for anything the meal voucher covers and that's minimally edible. At that exact moment, the financial alert fires: someone ran a colossal unfiltered process in the cloud. The cost projection went up US$ 1,400 in two hours. In 2011, we'd blow out the server's memory; today, we blow out the company credit card. I get my stun taser ready — which HR now calls a "feedback meeting."

11:59. No one breathes. Every status about to flip to the hamburger icon (🍔) at once. And there I am, investigating who's overloading the database: a third-party service fetching the same blessed row ten times a second. Fifteen years later. The very same flaw. Only now it's running in the cloud.

13:05. I get back from lunch and run straight into the classic request: "Can you run a tiny tweak directly on my time-clock record?" In 2011, I'd say I'd "think about it." Today, I explain that the system is in the cloud, has an immutable audit trail, and LGPD protection. Then I say I'll think about it.

13:20. Time to set up the data cleanup policy. There are 40 terabytes of useless files piled up since 2009 that "we'll need someday" — it's the reincarnation of my 2011 boss in the form of a retention policy. The difference is that now the "keep everything" mindset shows up every month on the AWS bill, charged in dollars. Cold cloud storage is our new lizard closet.

13:45. Nap Time, remote edition. Three cameras go dark simultaneously after lunch, followed by a "Can you repeat that? You cut out here" that fools no living soul. In person, the sport is still seeing who takes the first photo of whoever fell asleep in their chair.

14:00. "Strategic alignment" meeting with the client's IT. I chase down the firewall rule and the necessary access. In return, they congratulate me for discovering that their bottleneck was the antivirus freezing the database. Five minutes later, the meeting dies, swallowed by cell phone notifications. Google Calendar reschedules everything by itself for next week — hey, at least that got better.

14:30. A client analyst asks me to review his "data architecture": it's 300 spreadsheets stitched together by VLOOKUPs, feeding a dashboard that freezes every single Monday. When I mention best practices and data contracts, he makes the face of someone who's seen a ghost. Some IT classics never age.

14:52. The client's manager calls in a panic, chewing his dessert: the board demands, by tomorrow, a prototype "on top of our data." No one has ever built something like that in one afternoon. I recycle some old code, patch together three data sources, and stand up a dashboard that, if you squint, almost looks like high-level strategy. I spend the rest of the day trying to explain why that pretty number needs to be reviewed before it reaches the board. That part, no shortcut can fix.

15:20. The client's connection drops and the authentication system decides not to send the token. At the first sign of general instability, a chessboard, a deck of cards, and — I swear to God — a harmonica materialize in the office. I take the chance to chase down my monitor ticket for the fourth time and spend 15 minutes explaining to the vendor that "a backup strategy without testing the restore isn't a backup, it's just faith." I've been repeating that same sentence since 2011.

16:00. The client's manager enters the virtual crisis room sending a GIF of someone punching a desk. Miraculously, the network comes back two minutes later.

16:20. A truly critical ticket: the payroll closing system froze. "If it doesn't come back, no one gets paid and it's IT's fault." The analyst responsible for building it left the company; whoever inherited the mess has no idea what the code does. Documentation does exist: it's beautiful, flawless, but it describes a version of the system that hasn't existed since March. I map the code's dependency network like a detective hunting for clues and, voilà: a table join multiplying rows and freezing everything.

17:00. Fixed, tested, and shipped. The end user's response: "Good thing IT fixed it this time." One more happy user, completely unaware that, between his click and the number on the screen, there are fifty integrated systems, two billion daily events, and a data engineer emotionally held hostage by a server named Lady Aracnea II.

17:05. The manager announces: the board canceled the prototype presentation it demanded so urgently. The decision was made five minutes ago. The overtime goes into the mythical "hour bank," to be "negotiated in the future." In silence, a few analysts turn on the "Open to Work" tag on LinkedIn. I go back to my data cleanup.

17:40. I update the task board with my fingers crossed. A wave of melancholy hits me when I realize it's been ten days since I closed the laptop before 7 p.m.

17:55. Everyone already on "leaving 👋" status when the on-call siren sounds. No one breathes. No one looks. But the rotation is mine: the region where the client's database runs went down, and tomorrow is a billing peak day. While my automated code tries to rebuild the infrastructure from scratch, I watch the green Slack dots go out one by one, each accompanied by a parting joke about "the hard life of the person carrying the pager."

19:35. Mission accomplished. Infrastructure rebuilt, standby database promoted, backup validated, permissions restored, and tests run. All in an hour and a half. In 2011, the same catastrophe would have cost me three hours in the middle of the night just looking for the installation CD. Yeah, I think I'm getting good at this.

I close the laptop and "head home": I take eight steps to the living room couch. At least parking is free now. The hour bank, as always, keeps being negotiated in the future.

Technology has changed everything: the magnetic tape became a temporal database, the physical server in the hallway became code in the cloud. But, at the end of the day, nothing has changed: the user still deletes the wrong row, the boss still wants the project done yesterday, and perfect documentation is still a myth — only now, a much better-formatted myth.