This post is my current “oh-my-opencode” setup: Ghostty as the terminal, opencode as the driver, and GitHub Copilot (via MCP) as the model backend.

If you’re the kind of person who likes “dotfiles, but for agents”, this is for you.

What I’m Building

• A terminal-first workflow (fast, keyboard-driven)
• Multiple agents with explicit model + “variant” choices
• A permission system that’s strict by default but not annoying
• MCP access to GitHub so the agent can do real work
• Optional tmux layout so I can keep context visible

Prereqs (macOS)

I’m assuming:

• macOS + a terminal you like
• GitHub Copilot access enabled in your account
• You’re comfortable with JSON config files

For installs, I generally use Homebrew when possible:

brew --version

If opencode is distributed differently by the time you read this (it changes fast), follow their official install instructions and come back here for the configuration.

Terminal

👻 Ghostty

I like Ghostty because it feels lightweight and stays out of the way. It’s the “good ergonomics, no nonsense” terminal I’ve been looking for.

Small Ghostty tip: on Mac keyboards without Page Up/Down, scrolling is still doable. I mostly use trackpad scrolling, but when I’m in keyboard mode, I bind keys in Ghostty to scroll (page/line) so I don’t break flow.

Opencode

The whole setup revolves around two files:

• opencode.json: opencode core settings (plugins, permissions, MCP)
• oh-my-opencode.json: “agent roster” + tmux + category routing

opencode.json

This is the “policy” file. It’s where I decide what the tool is allowed to do in my shell, what plugins are enabled, and which MCP servers it can talk to.

opencode.json

{
  "$schema": "https://opencode.ai/config.json",
  "plugin": [
    "oh-my-opencode@latest"
  ],
  "permission": {
    "bash": {
      "*": "ask",
      "sudo *": "deny",
      "npm *": "allow",
      "npm install *": "ask",
      "mvn *": "allow",
      "mvn install *": "ask",
      "nvm *": "allow",
      "npx *": "allow",
      "npx install *": "ask",
      "bun *": "allow",
      "bunx install *": "ask",
      "echo *": "allow",
      "./mvnw *": "allow",
      "./mvnw install *": "ask",
      "make *": "allow",
      "sqlfluff *": "allow",
      "cat *": "allow",
      "ls *": "allow",
      "head *": "allow",
      "tail *": "allow",
      "test *": "allow",
      "grep *": "allow",
      "true *": "allow",
      "which *":"allow",
      "find *": "allow",
      "mkdir *": "allow",
      "sort *": "allow",
      "sed *": "allow",
      "tee *": "allow",
      "base64 *": "allow",
      "ps *": "allow",
      "wc *": "allow",
      "jq *": "allow",
      "rg *": "allow",
      "jar tf *": "allow",
      "jar xf *": "allow",
      "javap *": "allow",
      "printf *": "allow",
      "rm -rf*": "ask",
      "az *": "ask",
      "gh *": "allow",
      "gh *delete*": "deny",
      "gh *DELETE*": "deny",
      "gh *Delete*": "deny",
      "gh org *": "deny",
      "gh secret *": "deny",
      "acli *": "allow",
      "acli *delete*": "deny",
      "acli *DELETE*": "deny",
      "acli *Delete*": "deny"
    },
    "external_directory": "ask",
    "doom_loop": "ask"
  },
  "mcp": {
    "github": {
      "type": "remote",
      "url": "https://api.githubcopilot.com/mcp/",
      "enabled": true,
      "headers": {
        "Authorization": "Bearer {file:./secrets/github-mcp-pat}"
      }
    }
  },
  "share": "disabled",
  "lsp": {
    "kotlin-ls": {
      "command": ["kotlin-lsp", "--stdio"],
      "extensions": [".kt", ".kts"]
    },
    "kotlin-lsp": {
      "command": ["kotlin-lsp", "--stdio"],
      "extensions": [".kt", ".kts"]
    },
    "markdown": {
      "command": ["marksman", "server"],
      "extensions": [".md"]
    },
    "jsonnet": {
      "command": ["jsonnet-language-server", "--stdio"],
      "extensions": [".jsonnet", ".libsonnet"]
    },
    "yaml": {
      "command": ["yaml-language-server", "--stdio"],
      "extensions": [".yaml", ".yml"]
    },
    "postgres": {
      "command": ["postgres-language-server", "lsp-proxy", "--stdio"],
      "extensions": [".sql", ".pgsql"]
    }
  },
  "command": {
    "create-pr": {
      "template": "Create a draft PR for the changes in the current branch. The PR should be created against the main branch. The PR title should be a concise summary of the changes, and the PR description should include a detailed explanation of the changes, the reasoning behind them, and any relevant context. If there are any specific areas of the code that require attention or review, please highlight them in the description.",
      "description": "Create a draft PR for the current branch"
    },
    "review-code": {
      "template": "Ask @oracle to do a code review. Create an actionable plan with your recommendations and suggestions based on the findings. Ask for my feedback and approval before updating any code based on the plan.",
      "description": "Do a code review with @oracle"
    },
    "review-pr-comments": {
      "template": "Use the receiving-code-review skill to review PR comments. IMPORTANT: Ask for feedback and approval before updating any code based on the received reviews. IMPORTANT: Always reply on the PR comment thread for all comments received AFTER feedback and approval and any fixes have been implemented.",
      "description": "Review PR comments"
    },
    "review-pr": {
      "template": "Ask @oracle to review the code introduced in PR $1. If any actionable issues are found, add one comment to the PR for each issue with a summary and possible discussion points for the PR author. When adding comments, always try to add them to the correct line(s) in the PR code diff.",
      "description": "Review PR"
    }
  }
}

Notes on permissions

My default is: ask on everything, then allow the boring read-only stuff.

• "*": "ask" is the “seatbelt”. It stops the model from doing something clever-but-wrong while I’m half-paying attention.
• I explicitly deny sudo and the obvious “oops” commands.
• I allow cat, ls, rg, jq, etc. because those are basically just looking.
• I keep package installs as ask even if the ecosystem command itself is allowed (npm install, mvn install, etc.).

The best part is you can tune this as you learn where your personal risk tolerance is.

MCP: GitHub Copilot

This bit is the reason I’m writing the post.

"mcp": {
  "github": {
    "type": "remote",
    "url": "https://api.githubcopilot.com/mcp/",
    "enabled": true,
    "headers": {
      "Authorization": "Bearer {file:./secrets/github-mcp-pat}"
    }
  }
}

I keep the token out of the config and in a file under ./secrets/.

Practical checklist:

1. Create a personal access token (PAT) that works for your needs.
2. Put it in secrets/github-mcp-pat (no newline if you can help it).
3. Add secrets/ to .gitignore.

If you’ve ever accidentally committed a token once, you only need to learn that lesson a single time.

LSPs (optional, but great)

I like having language servers available even in agent workflows. It’s the difference between “best effort guessing” and “the tool actually understands my YAML/Markdown/SQL/Kotlin”.

If an LSP binary isn’t installed, opencode won’t magically fetch it. On macOS I install these via Homebrew / npm depending on the server.

oh-my-opencode.json

This is the fun one. It’s where I define who my agents are and what kind of tasks they handle.

oh-my-opencode.json

{
  "$schema": "https://raw.githubusercontent.com/code-yeongyu/oh-my-opencode/dev/assets/oh-my-opencode.schema.json",
  "agents": {
    "sisyphus": {
      "model": "github-copilot/claude-opus-4.6",
      "variant": "max"
    },
    "oracle": {
      "model": "github-copilot/gpt-5.2",
      "variant": "high"
    },
    "librarian": {
      "model": "github-copilot/claude-sonnet-4.5"
    },
    "explore": {
      "model": "github-copilot/gpt-5-mini"
    },
    "multimodal-looker": {
      "model": "github-copilot/gemini-3-flash-preview"
    },
    "prometheus": {
      "model": "github-copilot/claude-opus-4.6",
      "variant": "max"
    },
    "metis": {
      "model": "github-copilot/claude-opus-4.6",
      "variant": "max"
    },
    "momus": {
      "model": "github-copilot/gpt-5.2",
      "variant": "medium"
    },
    "atlas": {
      "model": "github-copilot/claude-sonnet-4.5"
    }
  },
  "tmux": {
    "enabled": true,
    "layout": "main-vertical",
    "main_pane_size": 50,
    "main_pane_min_width": 100,
    "agent_pane_min_width": 60
  },
  "categories": {
    "visual-engineering": {
      "model": "github-copilot/gemini-3.1-pro-preview",
      "variant": "high"
    },
    "ultrabrain": {
      "model": "github-copilot/gemini-3.1-pro-preview",
      "variant": "high"
    },
    "artistry": {
      "model": "github-copilot/gemini-3.1-pro-preview",
      "variant": "high"
    },
    "quick": {
      "model": "github-copilot/claude-haiku-4.5"
    },
    "unspecified-low": {
      "model": "github-copilot/claude-sonnet-4.5"
    },
    "unspecified-high": {
      "model": "github-copilot/claude-sonnet-4.5"
    },
    "writing": {
      "model": "github-copilot/gemini-3-flash-preview"
    }
  }
}

Why multiple agents?

I used to keep “one good model” and call it a day. That works… until it doesn’t.

Different tasks want different tradeoffs:

• “Search the repo and summarize what you find” wants speed and low cost.
• “Make a multi-file refactor and don’t break anything” wants careful reasoning.
• “Write text that sounds like a human who’s been burned by production” wants a writing-leaning model.

So I split the roles and named them. The names are silly, but the effect is real: I’m more deliberate about which agent I’m asking.

Agents vs skills (the mental model)

This is the distinction that made the whole setup “click” for me:

• Agents are people (or at least personas): a named model + variant + a set of expectations.
• Skills are moves: repeatable procedures you can invoke without rewriting the same prompt every time.

If you’ve ever had a model do a great code review once and then completely forget what “good review” means the next day, you already understand why skills matter.

I treat agents like specialists (explore, review, write), and skills like the playbook they execute.

Skills in practice: commands and prompt templates

In my config, I implement “skills” in two places:

1. Custom commands in opencode.json

These are basically pinned prompts with a name.

• create-pr is my “make a draft PR with a decent description” skill.
• review-code is my “ask an agent for a review plan, don’t touch code yet” skill.
• review-pr is my “review PR $1 and leave actionable comments” skill.

The important part isn’t the text itself—it’s that I can re-run the same workflow across repos and get consistent output.

2. Referenced skills inside prompts

Notice this template line:

“Use the receiving-code-review skill to review PR comments …”

That’s the same idea: name the move. In practice it means the agent should follow a specific routine when consuming feedback (summarize, propose fixes, ask approval, then reply on the thread).

Routing: categories as a lightweight skill selector

The categories section in oh-my-opencode.json is another way of making skills real.

I can label a task as quick or writing and the system routes it to a model that’s a better fit. That doesn’t magically guarantee quality—but it reduces the odds that I ask a “fast” model to do “slow” work.

tmux layout

When tmux is enabled, I like a main vertical layout so I can keep:

• one pane for my shell / repo
• one pane for the agent output

It’s a small thing, but it prevents the “scroll up, lose your place, forget what the model suggested” loop.

A Quick Sanity Check

Once the configs are in place, I do a couple of boring checks:

• Run a safe command (ls, rg, cat) and confirm it doesn’t prompt.
• Run a risky command (rm -rf ...) and confirm it does prompt.
• Trigger something that needs GitHub MCP and confirm auth works.

If something fails, it’s almost always:

• PAT missing/invalid
• PAT file path wrong
• secrets/ accidentally committed/ignored incorrectly

Day-to-day Workflow

My baseline loop is:

1. Ask an agent to explore and summarize
2. Ask a “careful” agent to propose a plan
3. Approve changes incrementally
4. Keep permissions tight so I can trust the tool

This is also why I keep share disabled. I treat my terminal agent like a coworker sitting at my keyboard.

Vibe Code

This section is intentionally unfinished. I’m still figuring out what I want “vibe coding” to mean when the tooling is actually capable of touching the repo.

The current rule is simple: vibe is fine, but only after the boring safety rails are in place.

A Fresh Venue: Nova Spektrum

With Oslo Spektrum under renovation, JavaZone relocated to Nova Spektrum. Just a short 10-minute train ride from Oslo, the venue offered significantly more space, enhancing the attendee experience with larger, more accessible rooms and an open exhibition hall—a welcome change over previous years.

More Attendees, More Energy

The move didn’t slow momentum—JavaZone 2025 welcomed approximately 3,800 developers, a few hundred more than the previous year, proving that the excitement for this community-driven event remains strong.

Programming Highlights

With over 100 speakers, the conference brought together engineers, architects, and industry leaders exploring themes like Java development, cloud-native architectures, DevOps, security, and more.

A standout feature was the pre-conference workshop day, followed by two full days of lightning talks—quick, dynamic presentations packed with insights.

Notable Sessions:

• “Building Secure AI Agents with Quarkus LangChain4j” – A hands-on workshop guiding participants through AI agent development and security.
• “Local Development in the AI Era” – Explored strategies to run AI models and code assistants entirely on local machines using tools like Ollama and Podman.
• “Non-deterministic? No problem! You can deal with it!” – Tackled testing AI’s unpredictable nature using Quarkus, Testcontainers, and more.
• “Boost Developer Productivity and Experience: A Deep Dive into Quarkus” – Demonstrated how Quarkus accelerates development workflows with features like live coding, Testcontainers, native images, and AI integration.

A Community-First Approach

Organized entirely by volunteers from javaBin, JavaZone remains the biggest community-driven Java conference in Norway, consistently fueled by passion and devotion to the developer community since 2001.

Throughout the conference:

• Food service was seamlessly integrated into the exhibition hall, eliminating long lunch queues and allowing attendees to grab bites anytime between sessions.
• The Roman Empire theme (honoring the “Romerike” region) added a playful, immersive vibe—booths and food stalls were stylized accordingly, creating a festive atmosphere that tied together culture and tech.

Social & Fringe Events

JavaZone’s famed evening party included music, food, and community bonding—a perfect way to unwind and network. Lightning talks also made for a spontaneous and engaging way to share ideas between sessions.

JavaZone 2025 exceeded expectations—despite a venue change, it maintained its warm, inclusive spirit and scaled up with ease. With richer sessions, a refreshed venue, strong attendance, and standout community energy, the conference reaffirmed its place as a can’t-miss event for Java practitioners and enthusiasts.

Software & Productivity

• brew as package manager
• asdf as language & tools management
• VS Code as code editor with Copilot integration
• Bear as notes keeping
• Arc as browser
• Kagi as search engine
• ChatGPT as AI assistant
• Bruno as API client
• OrbStack as Docker desktop replacement, plus virtual machines
• Spotify as music streaming
• Apple ecosystem for mail, calendar, reminders, messaging, and more.

Degoogling

As can be seen, I am using Kagi, which delivers excellent results without any garbage or ads, instead of Google. I also use email with my own domain through Apple’s iCloud email system, which allows for custom domains, as well as calendar and other features, all outside Google’s domain. Unfortunately, I have to maintain my Google account to use YouTube and occasionally for sites that do not allow creating accounts with a username and password, but only through Single Sign-On (SSO).

This year’s gathering drew over 8,000 delegates—government officials, civil society advocates, technical experts, youth leaders, and private-sector stakeholders—who joined forces to address pivotal challenges. From Africa’s urgent connectivity and policy needs to inventive sessions like the “Tower of Babel Chaos,” where participants navigated language barriers, the IGF 2025 brought both critical issues and bold experiments to the fore.

In the posts ahead, I’ll dive into how IGF 2025 was organized.

How IGF 2025 Was Organized: A Multilayered Event Experience

Attending IGF 2025 in Lillestrøm (June 23–27) felt like being part of a living, breathing ecosystem of internet governance. Event organizers masterfully structured the gathering into complementary formats, ensuring rich learning, networking, and creative breakthroughs. Here’s how the forum came together:

🧩 Workshops & Open Forums

These were the heart of the program: 90-minute deep dives where experts, activists, and newcomers discussed everything from data governance principles to digital identities. Open Forums added another layer—90-minute sessions hosted by major organizations like ICANN or UNESCO, inviting debate on current work and Q&A.

⚡ Lightning Talks & Flash Sessions

Need a quick hit of inspiration? Lightning Talks (10–20 minutes) and Flash Sessions were the high-speed engines of IGF—concentrated bursts where fresh ideas, project updates, or research findings were shared with the audience.

🎙️ Main & High-Level Sessions

The Plenary Hall hosted flagship debates: high-level panels and focus sessions on topics like AI, equity, and the internet’s future. These main sessions framed the broader discourse for the week.

🤝 Dynamic & Best Practice Forums

In addition to workshops and main tracks, IGF 2025 featured Dynamic Coalition meetings—stakeholder-driven working groups focused on themes like IoT, child online safety, or gender and internet governance. There were also Best Practice Forums, where attendees shared successful models and real-world examples.

🌐 Networking Sessions & IGF Village

Beyond formal sessions, the event was infused with social energy:

• Networking rooms offered small-group dialogues on topics like resilience or youth inclusion.

• IGF Village featured physical and virtual booths from NGOs, tech projects, companies, and institutions—perfect for demos, informal chats, and spontaneous meetups.

  

🌍 Host Country & Community-Minded Events

IGF 2025 emphasized both global reach and local color:

• Host country open stages showcased Norway-led discussions on policy and digital public infrastructure.
• Social events—from opening receptions to a “Walk to the birth of the Internet” in Kjeller—added cultural depth, bonding the community in relaxed settings.

🧭 Hybrid Format & Inclusivity

In a nod to global inclusivity, every session integrated virtual participation—speakers and rapporteurs joined remotely, while on-site attendees engaged in real-time. The result: a truly hybrid experience that welcomed voices across time zones

Thanks for a great conference that also was free to enter.

The accessibility and openness of IGF 2025 made it a space where anyone—regardless of background or budget—could take part in shaping the future of the internet. That spirit of inclusion is exactly what digital governance needs right now: more voices, more perspectives, more collaboration.

Until next time—onward toward a more open, equitable internet.

To achieve this have Ruter startet a pilot project with self-driving vehicles in the Grorud Valley, Oslo. The purpose of the pilot project is to establish and test a small-scale self-driving transportation service and gain insights and experience from it.[2] This is the pilot project i’m a part of.

Winter in the Loop

What makes this pilot particularly interesting is the setting: Norway, in winter. Snow-covered roads, icy surfaces, freezing rain, low light levels, and unpredictable weather patterns create conditions that challenge even the most experienced human drivers. For autonomous vehicles (AVs), these challenges are multiplied.

Sensors such as lidar, radar, and cameras are sensitive to snow, slush, and fog. Lane markings may be obscured. Pedestrians are often bundled up and hard to detect. And road edges can disappear under several centimeters of white. Yet, this is exactly the kind of real-world testing that’s needed to make AVs viable in countries like Norway.

My First Car Ride

Using the autonomous vehicle service starts with the app Selvkjørende, which was only for beta testing through TestFlight. After downloading and opening the app, you’re welcomed by a clean interface that combines traditional public transport options with the new on-demand self-driving service.

On the home screen, you simply enter your destination. The app then checks for nearby available autonomous vehicles and shows you the estimated wait time. In my case, I selected Stovner Senter as my pickup location and was matched with a NIO ES8 vehicle. The app estimated a 7-minute wait, which was far from accurate. It was more like 20 minutes.

The map interface is detailed and responsive, allowing you to see your location, the approaching vehicle, and the planned route. I was picked up near Haugenstua Station, and the app showed real-time updates on the vehicle’s progress toward Stovnertårnet. Both pickup and drop-off points were clearly marked on the map, and the app even displayed how long the vehicle would wait at the pickup location.

During the ride, I could follow the car’s movements on the map in real-time. The interface was smooth, with clear communication about estimated arrival times and vehicle ID. The vehicle itself was quiet and comfortable, and a safety operator was present throughout the journey. It felt futuristic, yet surprisingly normal.

This first ride gave me a sense of how intuitive the user experience is. Everything from ordering the ride to completing the trip was designed to be as seamless as using any other ride-hailing app.

The Passenger Experience

Inside the car, each passenger has a front-facing screen mounted to the seat in front of them. This display shows the vehicle’s current location, speed, nearby traffic, and remaining time to drop-off — similar to what you’d see in a robotaxi concept. At one point during the ride, the display indicated “8 minutes to Stovnertårnet”, and I could follow our lane position and surrounding vehicles visually.

Despite the car driving itself, it felt very safe. The safety operator sat in the driver’s seat ready to take over if needed, but the vehicle handled everything on its own: navigating traffic, stopping at intersections, and making smooth turns.

A sticker in the car reminded us not to disturb the safety operator — an important detail that subtly reinforces the experimental nature of the ride. Still, from a passenger’s point of view, the entire journey felt well thought-out and user-friendly.

This first ride gave me a sense of how intuitive the user experience is. Everything from ordering the ride to completing the trip was designed to be as seamless as using any other ride-hailing app — but with a significant twist: no human driver behind the wheel - or maybe that will be possible in the summer.

Obstacles

In progress..

The fascination with AI is understandable. The technology holds incredible promise across various domains. In healthcare, AI-driven personalized medicine could tailor treatments to individual patients, improving outcomes and reducing side effects. AI can also assist in diagnosing diseases, analyzing medical images with a level of precision that surpasses human capabilities. Imagine a world where early detection of diseases like cancer is possible through routine scans, significantly increasing survival rates.

In the realm of transportation, self-driving cars promise to reduce accidents, ease traffic congestion, and provide mobility solutions for the elderly and disabled. Companies like Tesla and Waymo are at the forefront of this transformation, with ongoing trials and promising advancements. Autonomous vehicles could revolutionize urban planning, reduce carbon emissions, and transform the logistics industry.

AI-powered tools in business can automate repetitive tasks, provide insights from vast amounts of data, and even help in decision-making processes. This could lead to unprecedented productivity boosts and cost savings. Businesses can use AI to forecast demand, optimize supply chains, and personalize customer experiences, leading to higher efficiency and satisfaction.

In our daily lives, AI is poised to become an integral part of our routines. From smart home devices that learn and anticipate our needs to virtual assistants that manage our schedules, the applications are endless. Imagine waking up to a home that adjusts the temperature to your preference, prepares your morning coffee, and organizes your daily tasks, all autonomously.

The excitement around AI is not just confined to tech enthusiasts. Investors are also caught up in the frenzy. Nvidia’s record profits are a testament to this enthusiasm, with its GPUs (graphics processing units) being a critical component in AI research and development. Companies across various sectors are investing heavily in AI, driving up valuations and stock prices. The stock market’s response to AI advancements mirrors the dot-com bubble of the late ’90s, characterized by rapid growth and soaring expectations.

However, beneath the surface of this hype lies a series of concerns that we must address to avoid a potential crash. AI has shown remarkable capabilities, but it’s not a panacea. The technology still has significant limitations, and in many cases, the hype outstrips the reality. For instance, fully autonomous vehicles are still years away from being a common sight on roads due to technical and regulatory challenges. Overpromising can lead to unrealistic expectations and eventual disillusionment.

The deployment of AI raises critical ethical questions. Issues like job displacement, data privacy, and biased algorithms need careful consideration. The enthusiasm for AI must not overshadow the need for responsible and ethical development and use. The potential for AI to reinforce existing biases and inequalities is a significant concern that requires proactive measures.

The AI boom has led to inflated valuations and expectations. If AI technologies fail to meet these lofty promises in the near term, we could see a significant market correction, impacting investors and companies alike. The financial instability caused by a sudden burst of the AI bubble could have far-reaching economic consequences.

As AI systems become more integrated into our lives, ensuring their safety and reliability is paramount. Regulatory frameworks are still catching up with the pace of AI development, leading to potential risks. The lack of standardized regulations can result in inconsistencies and vulnerabilities in AI applications.

To avoid the pitfalls of the AI hype train, we need a balanced and measured approach. It’s essential to temper expectations with a realistic understanding of AI’s current capabilities and limitations. This will help prevent disillusionment and maintain steady progress. Educating stakeholders about what AI can and cannot do is crucial for setting achievable goals.

Prioritizing ethical considerations in AI development can mitigate risks related to bias, privacy, and job displacement. Transparency and accountability should be at the forefront of AI innovation. Ethical AI practices include fair data usage, unbiased algorithm training, and inclusive representation in AI research.

Establishing robust regulatory frameworks will ensure the safe and responsible deployment of AI technologies. Collaboration between governments, industry, and academia is crucial in this regard. Regulations should focus on protecting consumers, ensuring data privacy, and promoting transparency in AI operations.

The field of AI is rapidly evolving. Continuous learning and adaptation are necessary to stay ahead of potential challenges and leverage new opportunities. Investing in research and development and fostering a culture of innovation can drive sustainable progress in AI.

AI is poised to transform various industries and aspects of our lives. In healthcare and medicine, AI can revolutionize diagnostics, treatment planning, and patient care. Predictive analytics can help in early disease detection, while AI-powered robots can assist in surgeries. Personalized medicine, driven by AI, can offer tailored treatments based on genetic profiles.

In education, AI can personalize learning experiences, offering tailored content and feedback to students. Intelligent tutoring systems can adapt to individual learning styles, improving educational outcomes. AI can also assist in administrative tasks, allowing educators to focus more on teaching.

In finance and banking, AI can enhance fraud detection, automate trading, and provide personalized financial advice. Robo-advisors can manage investment portfolios, while AI-powered chatbots can improve customer service. Predictive analytics can assist in risk management and financial forecasting.

AI can help in monitoring and managing natural resources, predicting environmental changes, and optimizing energy consumption. AI-driven solutions can support conservation efforts and combat climate change by analyzing vast amounts of environmental data.

In manufacturing and industry, AI can optimize production processes, improve quality control, and enhance supply chain management. Predictive maintenance can reduce downtime and increase efficiency. AI-powered robots can perform complex tasks, augmenting human capabilities.

The AI hype train is an exciting journey, filled with promise and potential. However, it’s crucial to navigate this path with caution, ensuring that we address the ethical, social, and technical challenges that lie ahead. By maintaining a balanced perspective and prioritizing responsible development, we can harness the power of AI to create a future that benefits everyone — without risking a devastating crash.

While the AI revolution offers remarkable opportunities, it also demands a measured and thoughtful approach. We must embrace AI with realistic expectations, ethical considerations, and robust regulations to ensure that this powerful technology serves humanity’s best interests. By doing so, we can ride the AI hype train towards a destination of sustainable progress and collective prosperity, avoiding the derailment that unchecked exuberance might bring.

What is dbt?

Dbt, short for Data Build Tool, is a command-line tool that enables data analysts and engineers to transform data in their warehouse more effectively. It allows you to write modular SQL queries, version control your transformations, and document your data in a consistent, reproducible manner. Dbt empowers data teams to own the entire analytics engineering workflow, from transforming raw data to creating clean, actionable insights.

Core Concepts of dbt

Before we delve into the project structure, let’s cover some core concepts:

Models

In dbt, a model is simply a SQL file that contains a select statement. Models are the building blocks of your dbt project. Each model file typically represents a transformation step, from raw data to clean, transformed data ready for analysis.

Sources

Sources refer to the raw data tables in your database. Defining sources helps you keep track of the origin of your data and ensures consistency in how you refer to them across your project.

Seeds

Seeds are CSV files that you can load into your data warehouse. These files can be used to manage reference data, small lookup tables, or any static data you need in your transformations.

Tests

Tests in dbt help you validate your data. You can write custom SQL tests or use built-in tests to ensure data quality and integrity. Common tests include checking for null values, uniqueness, and referential integrity.

Documentation

Documentation is an integral part of dbt. You can document your models, sources, and tests using YAML files. Dbt also provides a web-based interface to view your documentation and understand your data lineage.

Materializations

Materializations determine how dbt will build your models in the data warehouse. Common materializations include views, tables, and incremental tables. Each has its own use case, depending on the size of your data and performance requirements.

Exploring dbt’s Project Structure

A dbt project is organized in a specific structure to facilitate modular, scalable, and maintainable code. Here’s a typical layout of a dbt project:

dbt_project/
│
├── models/
│   ├── marts/
│   │   ├── finance/
│   │   │   ├── fct_orders.sql
│   │   │   ├── fct_revenue.sql
│   │   └── marketing/
│   │       ├── fct_campaigns.sql
│   │
│   ├── staging/
│   │   ├── stg_customers.sql
│   │   ├── stg_orders.sql
│   │
│   └── intermediate/
│       ├── int_customers.sql
│       ├── int_orders.sql
│
├── seeds/
│   ├── countries.csv
│   ├── currency_rates.csv
│
├── tests/
│   ├── unique_orders.sql
│   ├── not_null_customers.sql
│
├── macros/
│   ├── my_macro.sql
│
├── snapshots/
│   ├── customers_snapshot.sql
│
├── analyses/
│   ├── customer_lifetime_value.sql
│
├── docs/
│   ├── index.md
│
└── dbt_project.yml

Key Components

1. Models: This is where you define your SQL transformations. They are typically organized into subfolders like marts, staging, and intermediate to represent different stages of your data pipeline.

2. Seeds: These are CSV files that dbt loads into your warehouse. They can be found in the seeds folder.

3. Tests: Tests help ensure data quality. You can define them in the tests folder.

4. Macros: Macros are reusable SQL snippets or functions. You define them in the macros folder.

5. Snapshots: Snapshots track changes to your data over time. They are useful for slowly changing dimensions and can be found in the snapshots folder.

6. Analyses: This folder contains ad-hoc queries or analyses that you want to perform. They don’t typically contribute to your data pipeline but are useful for data exploration.

7. Docs: Documentation files go here. dbt will render these documents in a web-based interface for easy access.

8. dbt_project.yml: This is the configuration file for your dbt project. It defines the project’s settings, such as model materializations, seeds, and more.

Getting Started with dbt

To get started with dbt, follow these steps:

1. Install dbt: Install dbt using pip:

pip install dbt

2. Initialize a dbt Project: Create a new dbt project by running:

dbt init my_project

3. Configure Your Project: Edit the dbt_project.yml file to set up your project’s configurations.
4. Create Models: Add your SQL transformation files in the models folder.
5. Run Your Project: Execute your dbt project by running:

dbt run

6. Test Your Models: Validate your transformations by running:

dbt test

7. Generate Documentation: Create documentation for your project by running:

dbt docs generate

8. View the documentation with:

dbt docs serve

Conclusion

Dbt is a powerful tool for data transformation and management. By understanding its core concepts, exploring its project structure, and leveraging its key components, you can transform raw data into actionable insights more effectively. Remember, dbt encourages a modular, iterative workflow, making your data transformations more maintainable and scalable.

Tune in for the next dummies post coming soon. If you also stick around for my non-dbt related writing, I appreciate you. It doesn’t always feel right to promo those posts on LinkedIn, so I am grateful for those of you that read them anyways.

Subscribe to the RSS feed, tell your friends, you know the drill. Thanks for reading.

Enhanced Tracking Efficacy

One pivotal aspect to highlight is the heightened efficacy of tracking afforded by self-hosted analytics. With an increasing number of internet users employing ad blockers, approximately 26% of the online populace, the likelihood of obtaining analytics data for individual users is significantly higher with self-hosted solutions. By hosting analytics on dedicated subdomains such as analytics.yourdomain.com, the probability of encountering script blocking is notably diminished.

Ideological Integrity: Customer versus Lead

A profound ideological shift underpins the migration towards self-hosted analytics, epitomizing a transition from being perceived as a lead to asserting one’s status as a valued customer. Google Analytics, with an imposing 73% market share, commands a substantial presence in the web analytics landscape. However, the symbiotic relationship between Google Analytics and Google’s advertising services underscores a fundamental reality: users are not merely customers but potential leads. This paradigm shift not only underscores a commitment to data privacy but also ensures a more equitable transactional relationship, fostering trust and integrity in analytics practices.

Moreover, from a data privacy standpoint, self-hosting offers unparalleled control and ownership over visitor data. Alleviating concerns surrounding the extent of data accessibility by third-party entities, self-hosted analytics empower website owners with the autonomy to safeguard sensitive information, reinforcing compliance with stringent data protection regulations.

Embracing Simplicity and Efficiency

Simplicity, often overshadowed by the allure of complexity, emerges as a guiding principle in the selection of analytics solutions. While Google Analytics boasts a plethora of advanced features catering to diverse analytical needs, the inherent complexity may overshadow its utility for many users. Opting for a self-hosted analytics solution transcends the allure of convoluted functionalities, prioritizing a streamlined user experience tailored to address core analytical requirements. A discerning focus on traffic volume, source attribution, and page engagement underscores the pragmatic approach adopted in the deployment of analytics solutions.

Unveiling My New Analytics Framework

In embracing the ethos of self-hosted analytics, the adoption of Ackee emerges as a testament to a professional commitment to privacy, efficiency, and reliability. While the setup process may necessitate a marginally longer time investment relative to conventional alternatives, the dividends reaped in terms of data sovereignty, and analytical fidelity far outweigh the initial setup complexities. Ackee’s open-source architecture coupled with its intuitive user interface exemplifies a confluence of technological innovation and user-centric design, culminating in a superlative analytics experience that transcends conventional paradigms.

In essence, the transition to self-hosted analytics epitomizes a professional evolution towards data autonomy, ideological integrity, and operational efficiency, setting a precedent for conscientious analytics practices in the digital era.

The world of competitive chess has witnessed a seismic shift in recent years. The traditional clash between human grandmasters has now been joined by an unexpected contender: artificial intelligence. As machines continue to evolve, the question looms large: Is this the end of an era for human supremacy in chess?

In this article, we delve into the fascinating rivalry between two exceptional players: World Champion Magnus Carlsen and rising star Hans Niemann. Their encounters on the board have captivated fans, sparked controversies, and raised fundamental questions about the future of the game.

The Rise of Magnus Carlsen

The Chess Prodigy

Magnus Carlsen burst onto the international chess scene like a comet. His prodigious talent was evident from an early age, and he quickly climbed the ranks. His intuitive understanding of positions, relentless work ethic, and unyielding will to win set him apart. In 2013, he achieved the ultimate pinnacle: becoming the World Chess Champion.

The Reign of Carlsen

Carlsen’s reign has been marked by a unique blend of creativity, precision, and psychological warfare. His mastery of endgames, ability to grind out wins, and uncanny intuition have left opponents bewildered. His famous “Carlsen grind” is a testament to his tenacity—a slow, methodical squeeze that breaks even the most resilient defenses.

The Machine Challenge

But then came the machines. Deep Blue, Stockfish, AlphaZero—the silicon adversaries that threatened to dethrone human champions. Carlsen faced these digital opponents head-on, participating in man-versus-machine matches. His resilience was admirable, but the gap between human and machine seemed to narrow inexorably.

The Emergence of Hans Niemann

The American Prodigy

Enter Hans Niemann, a young American with fire in his eyes and a hunger for victory. Niemann’s rise has been meteoric. His tactical acumen, fearless play, and audacity against established grandmasters have turned heads. And it was during the 2022 Sinquefield Cup that he etched his name into chess history.

The Sinquefield Sensation

In a matchup against none other than Magnus Carlsen, Niemann caused seismic waves. The world champion, perhaps underestimating his opponent, found himself outplayed. Niemann’s smooth maneuvers and unwavering clock management left Carlsen scrambling. And when the chance arose, Niemann seized it—a victory that reverberated through the chess world.

The Controversy

Carlsen’s withdrawal from the Sinquefield Cup after losing to Niemann sparked controversy. Was it a tacit accusation of cheating? The chess community debated, but Niemann remained undeterred. His audacious play had disrupted the status quo, and the machine-human balance trembled.

The Endgame

A New Paradigm

As we witness the clash of titans—Carlsen’s experience against Niemann’s fearlessness—we realize that chess is evolving. Machines analyze billions of positions per second, but humans bring intuition, creativity, and emotion to the board. Perhaps the future lies in a harmonious blend—a symbiosis where humans guide the machines, and machines elevate human play.

The Final Move

So, is this truly the end for competitive chess as we know it? Not quite. The battle rages on, and the pieces dance across the 64 squares. Magnus Carlsen and Hans Niemann—their names etched in chess lore—are the vanguards of this new era. As the clock ticks, we await the final move—the one that will define the future of our beloved game.

Transform the data

The 3-axis raw signals tAcc-XYZ and tGyro-XYZ from the accelerometer and gyroscope were used to create this data set. At a constant rate of 50 Hz, the time domain signals (prefix ’t’ to signify time) were collected. To reduce noise, they were filtered with a median filter and a 3rd order low pass Butterworth filter with a 20 Hz corner frequency. Using a low pass Butterworth filter with a corner frequency of 0.3 Hz, the acceleration signal was split into body and gravity acceleration signals (tBodyAcc-XYZ and tGravityAcc-XYZ).

The 3-axis signals tAcc-XYZ and tGyro-XYZ depicted in Figure 1 and 2 are the raw data from the accelerometer and gyroscope.

To reduce noise, it where filtered with a median filter, as shown in Figures 3 and 4, and a 3rd order low pass Butterworth filter with a 20 Hz corner frequency, like shown in Figures 5 and 6. In both acceleration and gyroscope data, a median filter with a filter length of 5 was employed.

Another low pass Butterworth filter with a corner frequency of 0.3 Hz was used to split the acceleration signal into body and gravity acceleration signals (tBodyAcc-XYZ and tGravityAcc-XYZ), like shown in Figures 8 and 7. And separating acceleration signal into body and gravity acceleration signals by each window, like shown in Figure 9.

To acquire Jerk signals, the body linear acceleration and angular velocity were calculated in time (tBodyAccJerk-XYZ and tBodyGyroJerk-XYZ), as shown in Figure 10 and 11.

Lastly a Fast Fourier Transform (FFT) was applied to some of these signals producing fBodyAcc-XYZ, fBodyAccJerk-XYZ, fBodyGyro-XYZ, fBodyAccJerkMag, fBodyGyroMag, fBodyGyroJerkMag, as shown in Figure 12. The ‘f’ is to indicate frequency domain signals.

See DiFronzo/LSTM-for-Human-Activity-Recognition-classification for access to the repository with all the data.

Early life and education

Aksel Waldemar Johannessen was born October 26, 1880 in Christiania (today Oslo), Norway. He grew up at Kampen, where his father, Julius Johannesen, was a shoemaker and auxiliary policeman. He was connected early to the artist community at Hammersborg behind the Trinity Church. In the period of 1900 and 1907, he become educated as an artist, including seven semesters at the Norwegian National Academy of Craft and Art Industry, the last years with Lars Utne in the sculptor class. He learned the art of woodcarving in the workshop of the internationally famous Johan Borgersen.

Work in Gjøvik

Johannessen married Anna Gjermine Nilsen in 1907, and their relationship resulted in two daughters. They moved to Gjøvik the same year, where he worked as a cabinetmaker and woodcarver in the company Edvard Sandvold. He lived at Sandvoldgård in Welhavens street. Johannessen did not paint at this time, but he made a number of sculptures. They lived in Gjøvik until 1912.[1]

Returning to Kristiania

The couple moved back to Kristiania in 1912, he and his wife Nilsen came into close contact with the theater community and the social circle around Hulda and Arne Garborg. The couple founded a dressmaking shop named Heimen Teiknekontor. It became a permanent sewing room for the Norwegian Theatre, which was founded in 1913, and Johannessen became a stage worker for the theater. He participated in the infamous theater group after the first performance: “Jeppe paa Berget” by Ludvig Holberg. In Heimen’s sewing room, the first interest in the Norwegian folk costumes also began to emerge, which led to the reconstruction of the Valdres costume/bunad and the Gudbrandsdal costume/bunad. Johannessen began painting in 1912, although his work as a painter remained for the most part unnoticed by those around him. Throughout his life, Johannessen seemed to distance himself from the rather bourgeois and snobbish artistic community.

Death

At times had Johannessen alcohol problems, and from 1918 to 1921 was he in a very rough period. An attack of pleuresy in his youth had left him in delicate health. In 1922, was his wife, Nilsen, sick with incurable cancer. In sheer despair was Johannessen unable to control his alcohol consumption. He was found in the gutter of an alley, driven to Ullevål University Hospital, where the terminally ill Nilsen was sitting next to him when he died of pneumonia at the age of 42 on October 25, 1922.

At the time of Johannessen’s death, his wife Nilsen had only four months left to live. During this brief space of time, his wife set about to tidy up both her own life and that of her husband. Assisted by Hulda Garborg, she saw to it that her two daughters were well taken care of, and she sold the company “Heimen” to Bondeungdomslaget (the Society of Farm Youth) in 1923 for a purchase sums of about NOK 25,000, which at that time corresponded to 12-13 annual salaries.[2] She also contacted the artist Ola Abrahamsson, showing him the ninety works that Aksel Waldemar had painted, without making this known to anyone.

Memorial exhibition

The painter Ola Abrahamsson made sure that the about 90 leftover canvases of partly large format were shown at a memorial exhibition at Blomqvist in January 17, 1923.

The exhibition of these works of art created an overwhelming response with magnificent reviews, although a number of people did not comprehend the direct, realistic messages of his art. Jappe Nilssen, Norway’s foremost art critic of the time, the man first to recognize the talent of Edvard Munch and later to become his close friend, wrote what was probably the most positive review of his entire career: “… I can hardly recall having seen anything like this ever before in Nordic art.”

Munch was also full of enthusiasm: “this is something of the most extraordinary that I have ever seen”, and “finer works are not being painted to-day”. The sculptor Gustav Vigeland sent the widow some money. People were astonished to learn that Johannessen only gained prominence after his death. Myths were created about poverty, need, hunger and death in solitude, an echo from the Bohemian period of the 1880s. Johannessen’s canvasses were then subsequently packed and stored, and then went into total oblivion.

Thematically, the pictures are very ambitious, and range from the grotesque to the idyllic; from depictions of sexuality, violence, prostitution, alcoholism and war to idyllic and intimate works.

Legecy

The exhibition held 1923 in Blomqvist Gallery was reconstructed in the same gallery in 1992 with great success. The following year, 1993, the same exhibition was shown in the art museums of Stavanger, Skien, and Bergen. During these exhibitions, several foreign museums and galleries also discovered the artist and paid great interest to Johannessen. Under the supervision of the renowned art historian Professor Dr. Erich Steingräber, General Director of the Bayerische Staatsgemäldesammlung in Munich (Pinakothek) and honorable professor at Munich University, new exhibitions were arranged 1994 in Palazzo Te in Mantua and in the Doge Palace in Venice, and later in Castle Maretsch, Bolzano South Tirol. In 1995 exhibitions followed in Castle Lamberg, Steyr (Austria) and then finally in 1996 in Berlin.

International museums and institutions have acquired paintings by Johannessen, amongst them the famous collector and museum-founder Prof. Dr. Rudolf Leopold in Vienna.

Queen Elizabeth was supposed to open it all, but canceled due to illness. William, Kate and Charles, on the other hand, they arrived. And Barack Obama. As well as the world’s richest men, like Jeff Bezos and Bill Gates.

…and Greta Thunberg then. But she is at COP26 to protest.

In 2015, at COP21 in Paris, 197 parties agreed to establish a legal instrument that would govern climate change mitigation and adaptation efforts. This became known as the Paris Agreement. It includes overarching goals to keep global temperature rise below 2° C, with efforts to limit warming to 1.5 °C, and increase countries’ resilience to climate impacts. It also aims to ensure sufficient financing to achieve these targets. Like with most politic you only need to make an effort. I belive there should have been set a more strict goal.

So, do I agree with everything Thunberg is talking about? No, not everything. But Thunberg has some important points we should not throw under the couch. She is criticising the goals for cutting emissions which cause global warming, saying: “We don’t just need goals for just 2030 or 2050. We, above all, need them for 2020 and every following month and year to come”. Great! Now what action should be taken? That is a hard question to answer. Even Thunberg has avoided getting into the detail about that.

“We can no longer let the people in power decide what is politically possible. We can no longer let the people in power decide what hope is. Hope is not passive. Hope is not blah, blah, blah. Hope is telling the truth. Hope is taking action. And hope always comes from the people.” said Greta Thunberg

It remains to be seen how well the aims for COP26 will end up being addressed and if the effects of the conference will be stark enough to convince critics.

“It is a young life that has been extinguished, and I understand that he has meant a lot to many young people,” said Swedish Prime Minister Stefan Löfven.

Einár’s death sent a shock wave in Scandinavia and once again has the discussion regarding violence and gang-related activity within the hip hop community emerged. Many of Einár’s songs reference a life of crime, including drugs and weapons. Sweden has seen a rise in organised crime activity in recent years and several shootings have occurred in Stockholm, Göteborg, and Malmö.

Sweden’s Minister for Gender Equality and Housing Märta Stenevi twitted that “violence must be stopped”.

Ännu en ung människa har fallit offer för det dödliga våldet. Bakom varje rubrik finns en familj som förlorat ett barn. Jag tänker idag på alla dessa som förlorat en anhörig. Våldet måste stoppas.https://t.co/aqFnalxon5

— Märta Stenevi (@martastenevi) October 22, 2021

The motive of the shooting remains unclear but the police have not ruled out that the incident could be gang-related.

RIP Einàr you’ll always be remembered for your music and description of the contemporary criminal scene in Stockholm ❤

The two new chips look to change that situation, with Apple going all-out for performance, with more CPU cores, more GPU cores, much more silicon investment, and Apple now also increasing their power budget far past anything they’ve ever done in the smartphone or tablet space.

The M1 Pro: The Middle Child

The first of the two chips which were announced was M1 Pro – laying the ground-work for what Apple calls no-compromise laptop SoCs.

The company divulges that they’ve doubled up on the memory bus for the M1 Pro compared to the M1, moving from a 128-bit LPDDR4X interface to a new much wider and faster 256-bit LPDDR5 interface, promising system bandwidth of up to 200GB/s. That is most likely a rounded number, for an LPDDR5-6400 interface of that width would achieve 204.8GB/s.

The M1 Max: The Big Brother

Alongside the M1 Pro, Apple also announced a bigger brother – the M1 Max. While the M1 Pro catches up and outpaces the laptop competition in terms of performance, the M1 Max is aiming at delivering supercharging the GPU to a total of 32 cores. Essentially it’s no longer an SoC with an integrated GPU, rather it’s a GPU with an SoC around it.

The packaging for the M1 Max changes slightly in that it’s bigger – the most obvious change is the increase of DRAM chips from 2 to 4, which also corresponds to the increase in memory interface width from 256-bit to 512-bit. Apple is also advertising a massive 400GB/s of bandwidth, which if it’s LPDDR5-6400, would possibly be more exact 409.6GB/s. This kind of bandwidth is unheard of in an SoC, but quite the norm in very high-end GPUs.

Importing Modules/Packages

Before we start coding, import/install all of the following packages: NumPy, Pandas, and Sklearn.

# -*- coding: utf-8  -*-
import json
import numpy as np
import pandas as pd
import sklearn

from collections import defaultdict
from sklearn import linear_model

Loading in Our Data

Now it’s time for some data collection from Wikidata. For this example we are using the yearly (average) population stacked by country in a query (linked further down). This gives us a lot of interesting values and some with faults, unfortunately. I have chosen to filter this query to only include values from 2005 and newer. How you choose to import the query into the script is your decision. A passibility is to iterate over a SPARQL query by downloading the .rq file or just download a JSON file of the result from the query.wikidata.org site. Once you’ve downloaded the data set and placed it into your main directory (of the Python code) you will first need to clean the data, and later load it in using the pandas module.

Yearly Population stacked by country

# male/female population _must_ not be added unqualified as total population (!)
# this is an error and should be fixed at the item using P1540 and P1539 instead
# (wrong query result may be a manifestation of such)
SELECT ?year (AVG(?pop) AS ?population) ?countryLabel
WHERE
{
  ?country wdt:P31 wd:Q6256;
           p:P1082 ?popStatement .
  ?popStatement ps:P1082 ?pop;
                pq:P585 ?date .
  BIND(STR(YEAR(?date)) AS ?year)

  # IF multiple ?pop values per country per year exist, we prioritize by source
  #       census 1st, others 2nd, estimation(s) 3rd, unknown sources (none supplies P459) last
  # note: wikibase:rank won't help here: each year may have multiple statements for ?pop value
  #       rank:prefered is used for the best value (or values) of the latest or current year
  #       rank:normal may be justified for all of multiple ?pop values for a given year
  OPTIONAL { ?popStatement pq:P459 ?method. }
  OPTIONAL { ?country p:P1082 [ pq:P585 ?d; pq:P459 ?estimate ].
             FILTER(STR(YEAR(?d)) = ?year). FILTER(?estimate = wd:Q791801). }
  OPTIONAL { ?country p:P1082 [ pq:P585 ?e; pq:P459 ?census ].
             FILTER(STR(YEAR(?e)) = ?year). FILTER(?census = wd:Q39825). }
  OPTIONAL { ?country p:P1082 [ pq:P585 ?f; pq:P459 ?other ].
             FILTER(STR(YEAR(?f)) = ?year). FILTER(?other != wd:Q39825 && ?other != wd:Q791801). }
  BIND(COALESCE(
    IF(BOUND(?census), ?census, 1/0),
    IF(BOUND(?other), ?other, 1/0),
    IF(BOUND(?estimate), ?estimate, 1/0) ) AS ?pref_method).
  FILTER(IF(BOUND(?pref_method),?method = ?pref_method,true))
  # .. still need to group if multiple values per country per year exist and
  # - none is qualified with P459
  # - multiple ?estimate or multiple ?census (>1 value from same source)
  # - ?other yields more than one source (>1 values are better than optionally
  #                         supplied estimate, but no census source available)

  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" }
  FILTER(?year >= "2005")
}
GROUP BY ?year ?countryLabel
ORDER BY ?year ?countryLabel

Query found on Wikidata:SPARQL query service/queries/examples/advanced (shout-out to the person who made it, saved me a lot of time). Click here for full size image (CC0).

Now that we have cleaned the data and selected the interesting part of the query (country, year and population). We need to import the data into pandas. We also need, in this example, to flip the table (switch the place of column and row).

YEARS = ["2007", "2008", "2009" ,"2010", "2011", "2012", "2013"] # Years we are interested in


def getList(dict): # To get keys for the dict.
    list = []
    for key in dict.keys():
        list.append(key)

    return list

with open('query.json', 'r') as f: # Downloaded query in a JSON file.
    distros_dict = json.load(f)

allEntries = defaultdict(dict) # saves all the countries in the query with its data

for entry in distros_dict:
    allEntries[entry['countryLabel']].update({entry['year']: entry['population']})

selectedEnt = defaultdict(dict) # saves the countries in the query with its data that has all the values in the YEARS list

for country in allEntries:
    if all(elem in getList(allEntries[country]) for elem in YEARS):
        selectedEnt.update({country: allEntries[country]})

df = pd.DataFrame.from_dict(selectedEnt) # pastes it into pandas
data = pd.DataFrame.transpose(df) # flips the table

The data should now look something like this: print(data)

.                2007     2008     2009  \
Afghanistan    26349243 27032197 27708187
Algeria        35097043 35591377 36383302
... ... ... ...

Next it’s time to only select the data we want to use as test data, and remove the solution; in other words split the data. In this example I have choose to use population values from 2007-2012 (for the countries that have data for all the years between), where sklearn will do a prediction for 2013 (we also need the real value for this year).

data = data[YEARS]

predict = "2013"

Now that we’ve trimmed our data set down we need to separate it into 4 arrays. However, before we can do that we need to define what attribute we are trying to predict. This attribute is known as a label. The other attributes that will determine our label are known as features. Once we’ve done this we will use numpy to create two arrays. One that contains all of our features and another one that contains our labels.

X = np.array(data.drop([predict], 1)) # Features
y = np.array(data[predict]) # Labels

After this we need to split our data into testing and training set. We will use 90% of our data to train and the other 10% to test. The reason we do this is, so that we do not test our model on data that it has already seen.

x_train, x_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, test_size = 0.1)

Next is to implement the linear regression algorithm.

Implementing the Algorithm

We will start by defining the model which we will be using.

linear = linear_model.LinearRegression()

Next we will train and score our model using the arrays.

linear.fit(x_train, y_train)
acc = linear.score(x_test, y_test) # acc = accuracy

To see how well our algorithm performed on our test data we can print out the accuracy.

print(acc)

For this specific data set a score of above 80% is fairly good. This example has 99%.

Viewing The Constants

If we want to see the constants used to generate the line we can type the following.

print('Coefficient: \n', linear.coef_) # These are each slope value
print('Intercept: \n', linear.intercept_) # This is the intercept

Predicting the population in 2013

Seeing a numeric score value is nice but we would first like to see how well the algorithm works on a specific country. To do this we are going to print out all of our test data. Beside this data we will print the actual population in 2013 and our models predicted population value.

predictions = linear.predict(x_test) # Gets a list of all predictions

print("Country - sklearn guessed value for 2013, the Wikidata values (2007-2012), The Wikidata value (2013)")
for x in range(len(predictions)):
    for country in selectedEnt:
        if x_test[x][0] == selectedEnt[country][YEARS[0]] and x_test[x][1] == selectedEnt[country][YEARS[1]]: # To find the country used in the test data
            print(country, " - ", predictions[x], x_test[x], y_test[x])

Test result

0.999650607098148
Coefficient:
[ 0.41969474 -1.01050159 -0.20560013  0.0411049   1.3388236   0.41479332]
Intercept:
36691.20709852874

Full code

# -*- coding: utf-8  -*-
import json
import numpy as np
import pandas as pd
import sklearn

from collections import defaultdict
from sklearn import linear_model

YEARS = ["2007", "2008", "2009", "2010", "2011", "2012", "2013"]


def getList(dict):
    list = []
    for key in dict.keys():
        list.append(key)

    return list

with open('query.json', 'r') as f:
    distros_dict = json.load(f)

allEntries = defaultdict(dict)

for entry in distros_dict:
    allEntries[entry['countryLabel']].update({entry['year']: entry['population']})

selectedEnt = defaultdict(dict)

for country in allEntries:
    if all(elem in getList(allEntries[country]) for elem in YEARS):
        selectedEnt.update({country: allEntries[country]})

df = pd.DataFrame.from_dict(selectedEnt)
data = pd.DataFrame.transpose(df)

data = data[YEARS]

predict = "2013"

X = np.array(data.drop([predict], 1)) # Features
y = np.array(data[predict]) # Labels

x_train, x_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, test_size = 0.1)

linear = linear_model.LinearRegression()

linear.fit(x_train, y_train)
acc = linear.score(x_test, y_test)
print(acc)

print('Coefficient: \n', linear.coef_)
print('Intercept: \n', linear.intercept_)

predictions = linear.predict(x_test)

print("Country - sklearn guessed value for 2013, the Wikidata values (2007-2012), The Wikidata value (2013)")
for x in range(len(predictions)):
    for country in selectedEnt:
        if x_test[x][0] == selectedEnt[country][YEARS[0]] and x_test[x][1] == selectedEnt[country][YEARS[1]]:
            print(country, " - ", predictions[x], x_test[x], y_test[x])

Jupyter page (PAWS)

• Wikidata - Linear regression

This is my first time using this platform to write a blog post. I have not decided on what I’m supposed to write about. The platform is still a work in progress and was created since I’ve always wanted to start a technological blog. The topics will most likely be relevant to what I’m interested in or what I’m working on.

So, thank you for taking the time to read this first blog post; hopefully, there will be more to come in the future. Good-bye!