Cinnabar Notes

Building a Gateway VPS with OPNsense: Private IP, HAProxy, and WireGuard

Wed, 11 Feb 2026 00:00:00 GMT

[!NOTE] This post is about how I built a gateway VPS using OPNsense. This article does not cover technical details step-by-step but more about my experience and lessons learned. I hope you find it interesting.

I’ve always had a weird bias about FreeBSD vs Linux for server use.

FreeBSD just feels better, especially the TCP/IP stack.

Linux is great, but FreeBSD networking feels… like Ferrari

Anyway, I have three VPS on the same provider. Two of them are high spec (for me) and already running Linux doing real work. The third one is low spec and basically doing nothing. My provider also supports private virtual IP between VPS instances, so I had a cursed idea:

What if the idle VPS becomes the gate of everything?

Like a small edge router VPS that handles routing/firewall/VPN/reverse proxy, while the bigger Linux boxes just focus on workloads.

                           (Provider Private Network: 10.20.0.0/16)
        ┌───────────────────────────────────────────────────────────────────┐
        │                                                                   │
        │   [OPNsense VPS] 10.20.0.1                                        │
        │       |                                                           │
        │       |---> [Linux Server 1] 10.20.0.11  ---> (Services)          │
        │       |                                                           │
        │       |---> [Linux Server 2] 10.20.0.12  ---> (Services)          │
        │       |                                                           │
        │       |                                                           │
        │       └── WireGuard: wg0 10.60.0.1/24                             │
        │                    (Tunnel Network: 10.60.0.0/24)                 │
        └───────────────────────────────┬───────────────────────────────────┘
                                        │
                                        │  WireGuard tunnel (10.60.0.0/24)
                                        │
                                wg peer │ 10.60.0.2/24
                              ┌─────────▼─────────┐
                              │   [MikroTik]      │
                              │   LAN GW:         │
                              │   192.168.60.1/24 │
                              └─────────┬─────────┘
                                        │
                                        │ (Homelab LAN: 192.168.60.0/24)
                         ┌──────────────┴────────────────┐
                         │                               │
                 [Proxmox Host]                   [VMs on Proxmox]
               192.168.60.10                      192.168.60.101
                                                  192.168.60.102
                                                  ...

So I went down the FreeBSD firewall rabbit hole.

[!NOTE] My configuration has another trade-off: this gateway VPS is a single point of failure. If it goes down, everything goes down with it: reverse proxy, VPN, routing, the whole “gate of everything.” In my case, it’s acceptable because this is a homelab setup and I can afford some downtime. But in production, consider high availability (HA) setups.

OPNsense vs pfSense: I picked OPNsense for a lazy reason

I knew about pfSense/OPNsense since my undergrad era, my friend, Ahmad, was comparing router OS performance (pfSense, OpenWrt, etc.) for his thesis while I was benchmarking an Ethereum private network. We published our papers, Ahmad's paper and my paper here.

Now the funniest part is that I actually needed one of them.

When choosing OPNsense vs pfSense, I went with OPNsense (sorry Ahmad). Either because I did a deep philosophical comparison nor because I’m a firewall guru.

My brain literally went:

OPNsense is a pfSense fork → okay I’ll use OPNsense.

That’s it. Zero drama.

Easy Installation but Harder Access for First Time

OPNsense installation is chill, feels like installing FreeBSD. Then you reboot, you configure interfaces, you feel smart for 3 minutes and then the web UI is not accessible.

I had two NICs. I assigned WAN/LAN. Done. Then I tried to open the Web UI and… nope. I repeating the process a few times, checking IPs, interfaces, everything. Still no access.

At that moment I learned the classic lesson:

OPNsense Web UI is basically LAN-only by default and the firewall is not here to be your friend.

As a newbie, this combo is deadly. Because you can easily lock yourself out and then you start questioning life. Also I tried to just edit the config file like in Linux.

Spoiler: I didn’t even know where the web UI config file was. And even if I found it, I wasn’t confident editing it manually because it’s not that kind of system.

So yeah. First time OPNsense experience: I blocked myself.

My solution was very professional:

pfctl -d → finish setup in Web UI → pfctl -e

Yes, I temporarily turned off the firewall on a firewall OS, so I could configure the firewall. A peak engineering, but it worked, and after that I could proceed normally and tighten rules properly.

[!NOTE] Always have console access to your OPNsense box. If you mess up the firewall rules, you can always fix it from the console.

WireGuard: I suffered on FreeBSD… until OPNsense fixed my misunderstanding

Now the fun part: VPN.

I have a MikroTik router in my homelab managing all my internal VMs. Since MikroTik supports WireGuard now, I knew I could tunnel my homelab to the internet somehow. But here’s the thing: for a long time I had a wrong mental model.

I thought WireGuard “server” means you need an extra dedicated server just for tunneling. Like a separate box that only exists to accept connections and forward traffic. That idea alone already felt like more infra, more pain, more Docker.

So I kept avoiding the proper setup and relied on wg-easy because it was simple until I remembered: wg-easy = Docker again. Oh no.. Docker again.

Then I tried WireGuard inside OPNsense and suddenly it clicked:

I don’t need an additional tunneling server. OPNsense is the gateway, so it can be the WireGuard server too.

Once I understood the peer concept, everything made way more sense. WireGuard isn’t really “server vs client” the way I used to imagine it, it's basically peers exchanging keys, and one of them just happens to be the “hub” in my topology. After that, the setup felt weirdly smooth:

generate keys
define peers
set allowed IPs properly

The setup is smooth, but another challenge appeared: peers cannot ping.

As a noob, I thought that WireGuard (Group) is a rule for the WireGuard interface. But, I was wrong. First you should create interface first, so WireGuard real rules will appear under the correct interface (in my case, WG).

That part confused me for a while, but after I understood it, everything worked fine.

HAProxy: friendship ended with Nginx Proxy Manager

Next, reverse proxy.

Before this, I used Nginx Proxy Manager a lot because it’s easy: click-click-done, certificates, hosts, whatever. But it’s Docker-based and my gateway VPS was supposed to be clean and network-focused.

Then I noticed OPNsense can use HAProxy (and some people also use NGINX). I read a bit and my brain liked HAProxy because:

It’s very strong for TCP (Layer 4)
Load balancing feels native to it
It matches the gateway role better (for me)

NGINX is amazing, but HAProxy felt like:

“I exist to move traffic efficiently.”

So yeah:

Friendship ended with NPM. HAProxy is my best friend now.

…but also the friend that keeps giving you migraines.

Because unlike NPM where you just map domain → IP:port, HAProxy wants you to understand its whole worldview:

Frontends (where traffic enters)
Backends (where traffic exits)
Backend pools (multiple servers behind a backend)
Health checks (because HAProxy refuses to send traffic to something it thinks is dead)

Below here is comparison diagrams of both setups for two services: memo.cnbr.jp (hosted on Linux Server 1) and jellyfin.cnbr.jp (hosted on a VM in my homelab via WireGuard) if I used Nginx Proxy Manager vs HAProxy.

Nginx Proxy Manager

                        ┌─────────────────────────────┐
Internet / Clients ---> │ NPM (Docker) :443 / :80     │
 HTTPS SNI/Host         └──────────────┬──────────────┘
                                       |
                          Proxy Hosts (hostname routing)
                                       |
          +----------------------------+--------------------------------+
          |                                                             |
          v                                                             v
┌───────────────────────┐                                      ┌───────────────────────┐
│ memo.cnbr.jp          │                                      │ jellyfin.cnbr.jp      │
│ Forward to:           │                                      │ Forward to:           │
│ 10.20.0.11:3000       │                                      │ 192.168.60.101:8096   │
│ SSL: ON (Let's Encrypt│                                      │ SSL: ON (Let's Encrypt│
│ Health check: (lol)   │                                      │ Health check: (lol)   │
└───────────────────────┘                                      └───────────────────────┘


========================================================================================

HAProxy

                           ┌──────────────────────────────────┐
Internet / Clients  HTTPS  │ OPNsense HAProxy Frontend :443   │
 SNI/Host routing -------->│ Public Service "https-443"       │
                           └───────────────┬──────────────────┘
                                           |
                                   Rules (ordered)
                                           |
                 +-------------------------+--------------------------+
                 |                                                    |
                 v                                                    v
       Condition: hdr(host)=memo.cnbr.jp                    Condition: hdr(host)=jellyfin.cnbr.jp
                 |                                                    |
                 v                                                    v
        Action: use backend pool "bk_memo"                 Action: use backend pool "bk_jellyfin"
                 |                                                    |
                 v                                                    v
     Backend Pool "bk_memo" (10.20.0.0/16)              Backend Pool "bk_jellyfin" (via WG)
     server: 10.20.0.11:3000  (check?)                  server: 192.168.60.101:8096 (check?)
                 |                                                    |
                 v                                                    v
            (Services)                                           (Jellyfin VM)

And that last part… the health checks… is where my sanity started leaking.

The most evil moment is when everything looks fine, DNS is correct, certificates are okay, your service is running, but your browser says:

At one point, my Jellyfin suddenly wasn’t reachable. And because my setup involves OPNsense + HAProxy + WireGuard + MikroTik + firewall rules, my brain immediately went full paranoia mode:

“Did I break the WireGuard tunnel?”
“Is MikroTik routing wrong?”
“Is OPNsense blocking traffic?”
“Did I mess up NAT or firewall rules?”
“Is the private IP path dead?”
“Is my VPS provider doing something weird?”

So I started checking everything layer by layer like a confused network detective. Then… after way too long… I found the real reason:

HAProxy marked the Jellyfin backend as DOWN not because Jellyfin was dead but because the health check failed.

So HAProxy just refused to forward traffic and gave me 503, even though Jellyfin itself was actually fine. That moment was hilarious because I was debugging WireGuard + MikroTik + firewall like it was a deep network failure and the problem was literally:

“Health check says no.”

Since then I adopted my professional HAProxy workflow

Make the HAProxy route work first, skip health checks at the beginning, then enable health checks later

Conclusion

So yeah, that’s my experience building a gateway VPS with OPNsense. I’m still learning a lot, but so far it’s been fun.

My final setup looks great because it's cleanly separated:

OPNsense handles routing, firewall, VPN, and reverse proxy
Linux servers focus on workloads without worrying about networking

// END

Running Docker on FreeBSD

Sat, 15 Feb 2025 00:00:00 GMT

[!NOTE] This post is about how I run Docker on FreeBSD. I hope you find it interesting.

As a DevOps engineer, I often use Docker to containerize applications. Docker is a great tool for managing containers, and it makes it easy to deploy applications in a consistent and reproducible way. I have been using Docker for a long time, and I am very familiar with it.

Before I porting Nginx Proxy Manager to run on FreeBSD, I have to run it on Docker. I use it to manage my reverse proxies, and it makes it easy to set up SSL certificates and manage multiple domains. I have been using it on Linux, and I wanted to see if I could run it on FreeBSD. The bad news is Nginx Proxy Manager runs on Docker, and FreeBSD does not support Docker natively. So, I must find a way to run Docker on FreeBSD.

Running Docker on FreeBSD

The best way to run Docker on FreeBSD is to use a Linux virtual machine. I use bhyve as a hypervisor to run a Linux virtual machine.

+-----------+          +------------+          +----------------+
| FreeBSD   | -------- | bhyve      | -------- | Fedora CoreOS  |
+-----------+          +------------+          +----------------+
                                                        |
                                               +----------------+
                                               | Docker Engine  |
                                               +----------------+

The architecture is simple. I run a Fedora CoreOS virtual machine on bhyve, and I install Docker Engine on it. For easier installation, I use vm-bhyve to manage the virtual machine. Why Fedora CoreOS? Because it's designed for running containerized workloads. It's optimized for running containers, and it comes with Docker Engine pre-installed.

+-----------+          +------------------------+          +----------------+
| User      | -------- | Host                   | -------- | Fedora CoreOS  |
| 10.42.1.1 |          | 172.16.0.1:40000-50000 |          | 192.168.10.1:x |
+-----------+          +------------------------+          +----------------+

In my VPS, I only have 1 IPv4 address, so I allocate ports 40000-50000 from the host to the virtual machine. I use pf to configure port forwarding. Then, in Docker, I only port forward to the ports in range 40000-50000 in the virtual machine.

Managing Docker Containers

Originally, to manage the Docker containers, we must access the virtual machine with ssh or vnc. For easier management, I use portainer to manage Docker containers. Portainer is a lightweight management UI that allows you to easily manage your Docker containers, images, networks, and volumes. It's a great tool for managing Docker, and it makes it easy to deploy and manage containers. For testing, I run Ubuntu with XFCE4 desktop environment on Docker including xrdp for remote desktop and here is the result.

+-----------+---------+-----------+---------+-----------+
| FreeBSD   | Bhyve   | Fedora    | Docker  | Ubuntu    |
+-----------+---------+-----------+---------+-----------+

Known Bug

I have a problem with bhyve that the internet connection is too slow. The guest os always got terrible network internet speed (average in 2 Mbps) when the host internet speed average is 1 Gbps. I have tried using debian, alpine, and fedora coreos and it always got terrible network speed.

The solution is to disable tcp segmentation offload and tcp checksum offload in the host os. Disabling that in the host os will fix the problem, but will make the host speed is not maximum to max (fast enough). Here is the command to disable it.

ifconfig xxx -txcsum -rxcsum -tso -lero

That solution I found in this post.

Additional Notes

Running Docker on FreeBSD is not as straightforward as running it on Linux. But, with a little bit of effort, you can run Docker on FreeBSD using a Linux virtual machine. I have been using this setup for a while, and it works well for me. I can run Docker containers on FreeBSD and manage them easily using Portainer.

This setup is recommended for running Docker on FreeBSD? I can say, NO. I prefer to buy another VPS with Linux installed. That's the best way to run Docker. But, if you have a VPS with FreeBSD installed, you can use this setup to run Docker.

// END

My Journey with FreeBSD

Fri, 14 Feb 2025 00:00:00 GMT

This post is about my journey with FreeBSD, how I started using it, and my experience with it. I hope you find it interesting.

I first using FreeBSD in 2021, or perhaps even earlier, while experimenting on DigitalOcean in 2015/2016. At that time (2021), I used Google Cloud Engine as a platform to try it out. I cannot recall the exact date, but I distinctly remember using FreeBSD 12 back then. I eventually gave up due to the configuration files being different from those in Linux (which I will discuss later).

Why FreeBSD?

I remember when Debian deciding to use systemd as the default init system. I was confused and didn't like it. When Ubuntu forced me to use snaps, I was also confused. I wanted to use an operating system that respects my freedom and doesn't force me to use something I don't like. Surely, I hate that kind of thing. In an article, I have read that 10,000 lines code in systemd is just for init when in FreeBSD it's make a complete system. I was interested in that. FreeBSD pays great attention to the optimization of its code.

I have always been interested in UNIX-like operating systems. I have used Linux for a long time, and I wanted to try something different. I have heard a lot of good things about FreeBSD, such as its stability, security, and performance. I also wanted to learn more about the BSD family of operating systems. I also read an article that FreeBSD is more stable than Linux, and it is used in many critical systems. I was intrigued by the idea of using an operating system that is known for its reliability and security. I decided to give it a try and see how it compares to Linux.

My Experience

When I first started using FreeBSD, I was impressed by how easy it was to install. The installation process was straightforward, and I had no issues setting up my system. I was able to install all the software I needed using the package manager. But, I was a bit stressed because this operating system is different from Linux. I remember that I had to overnight to make touchpad, webcam, sound, and other things work. That's was a bit frustrating. I had to learn about ACPI, devd, and other things that I didn't know before, loading a kernel module, and so on.

I also had to get used to the different directory structure and file system layout. But, I like the structure of the FreeBSD system. I love the /usr/local/ directory, which is used for installing software from ports. I also like the /etc/rc.conf file, which is used for system configuration. I found it easier to manage system configuration files in FreeBSD than in Linux.

FreeBSD Handbook is a great resource for learning about FreeBSD. I have learned a lot from it. I also found the FreeBSD forums and mailing lists to be helpful and I love their jokes. The article from vermaden is also a great resource for learning about FreeBSD. I have learned a lot from his blog.

I also create a review of FreeBSD on my Youtube channel. But, it's in Indonesian. You can watch it.

What I Love About FreeBSD

FreeBSD is a real UNIX operating system. It is based on the original UNIX codebase and is POSIX compliant. What I mean here is that FreeBSD already encompasses the entirety of an operating system's needs. We know that Linux is essentially just a kernel, which is the core of an entire operating system. Whereas FreeBSD includes the entire operating system itself. So, FreeBSD is a complete operating system.

FreeBSD is UNIX-like because it is not certified by The Open Group as a UNIX operating system.

As someone who has experienced the stability of CentOS, I find FreeBSD to be even more stable. I remember when I use Debian 7, it's not stable like CentOS 6. I actually like Linux, but the frequently changing kernel versions make me uneasy. Additionally, while systemd has been helpful to me, it doesn't seem suitable for my cheapest, minimalist, and poor servers. Moreover, because FreeBSD is a real operating system, it is free from third-party interests that might have their own agendas. I still remember when the Linux community split into two: those who supported systemd and those who did not.

As someone who has used Linux, I feel that there are too many distributions. Many distributions, many tutorials, many documentations, many package managers, many configurations, many everything. I don't mind having many distributions, but please, come to an agreement to unify the configuration files. Especially now, there are distributions that store configuration files in different paths, which is quite confusing.

FreeBSD is essentially a community consisting of many people. Within FreeBSD, there is a core team made up of 10 individuals chosen by the community. This core team decides the direction of FreeBSD's development. Therefore, FreeBSD does not belong to a single person or company, but to the community.

Why democratic? Every commit to the Linux kernel must be approved by Linus Torvalds. This is not the case with FreeBSD. The core team is responsible for approving commits. This is a good thing because it means that the development of FreeBSD is not dependent on a single person. If Linus Torvalds is gone, who will approve the commits? This is a question that has been asked many times.

I still use Linux for certain purposes, such as in DevOps (docker). It must be acknowledged that Linux has advanced further than FreeBSD, especially in terms of hardware support and ease of use. I sometimes still grumble because ath10k still doesn't have a driver.

Now What?

Today, when I write that article, I use FreeBSD 14.2-RELEASE on my local NAS storage and my VPS. I have been using FreeBSD for a long time, and I have no plans to switch back to Linux except for AI and DevOps. I am happy with FreeBSD, and I have learned a lot from using it (even if I got a headache).

For my daily driver, I use MacOS for its simplicity and ease of use. In fact, MacOS contains a little bit of FreeBSD code. So, I can say that I am still using FreeBSD, but in a different form 😄.

I want to be a FreeBSD contributor, but I still have a lot to learn. I hope that I can contribute to the FreeBSD community in the future.

// END

Cheap Homelab Using Dell Optiplex 3050

Thu, 13 Feb 2025 00:00:00 GMT

I have been using a ZTE B860H (ARM based) for a long time as my Homelab. It's a great device, cheap and less power consumption. I have installed Kodi on it and use it as a media player, Jellyfin for streaming, and uTorrent for downloading. I want to say that ZTE B860H is good, but the main issue is the device encoding performance. It's not good for transcoding. So, I decided to build a new Homelab.

I have been thinking about building a cheap Homelab using a PC. My main PC is HP EliteDesk 800 G4 SFF that contains an Intel Core i7-8700, 16GB RAM, and 256GB SSD. I have a plan to use it as a NAS storage, after I bought a new PC. But, the build is too good and I love it (EliteDesk bro). I don't want to use it as a Homelab. So, I have been looking for a cheap PC for Homelab.

Finding a PC

I have been looking for a cheap PC for Homelab. I have a plan to buy a new PC, but I don't have enough money. I have been looking for a PC on the internet, and I found a Dell Optiplex 3050. It's a small form factor PC that contains an Intel Core i3-7100T, 8GB RAM, and 256GB SATA SSD. The price is around 70 USD (secondhand). I think it's a good deal, so I bought it without SSD. I got the real price is 55 USD.

The Dell Optiplex 3050 only have 1x 2.5" SATA slot. This is a bit of a problem because the main operating system must be installed in separate storage. I have a plan to use a 1TB HDD that I have for storage. So, I bought a cheap secondhand 128GB NVME SSD for the main operating system. The price is around 20 USD. The price is around 6.11 USD.

I don't think about how good the SSD is because it's just for running the operating system (just buy it). But, luckily the SSD is good. It's a UMIS 128GB. I have never heard about this brand before. But, I have read that this SSD used in Lenovo Thinkpad. So, I think it's good 😁.

Building the Homelab

I cannot say that I really build a Homelab or not because it just a PC with a 1 SATA HDD and low spec. The system is not RAID implemented.

For operating system, I use FreeBSD 14.2-RELEASE. I don't run Proxmox because high of loss performance and the processor is not good. I make 3 partitions on the 1TB HDD. The first partition is for Media (anime and movie), the second partition is for Data (cloud storage), and the third partition is for Secret (personal files) with just personal files partition encrypted with GELI. In other hand, the 128GB NVME SSD is used for the operating system. All of the partitions filesystem is ZFS.

The main application are installed inside jail. What I mean here is that I don't install the application directly on the host. I use bastille for jail management. I have installed Jellyfin, Radarr, Sonarr, Lidarr on same jail while I separate Nextcloud in different jail.

Result

I have installed Jellyfin for streaming. The encoding performance is good than ZTE B860H. The Kodi performance is also good. I can play UHD video without any problem (in ZTE B860H, I must waiting the video encoded first or buffering).

I also installed Nextcloud for cloud storage (and I love it). The main problem installing Nextcloud in FreeBSD more complex than in Linux. I have to install PHP, Postgres, Nginx, and Nextcloud itself including activating the PHP extension and configuring all of it.

I dont make my homelab as hosting server. I just use it for personal use. The main problem is internet connection in the dorm I live in is too bad (20 Mbps). For public access, I planned to use wireguard for VPN, connecting to my VPS.

Conclusion

Dell Optiplex 3050 is a good PC for Homelab. The price is cheap, and the performance is good enough for personal use with low power consumption than others. But, the main problem is the storage. The PC only have 1 2.5" SATA slot. For now, big HDD storage like 10 TB only available in 3.5" form factor. I'm sure that HP EliteDesk 800 G4 SFF is better than Dell Optiplex 3050 for homelab server (it contains 3 SATA slot and 2 M.2 slot).

My homelab build is enough for personal use. I can stream video, and cloud storage. The performance is good enough for me. I am happy with my homelab, and I have learned a lot from using it.

// END

JWT Implementation with Redis on Golang

Thu, 14 Mar 2024 00:00:00 GMT

For now, JWT (JSON Web Token) is a popular way to authenticate users in web applications. JWT is a token that is used to authenticate users and is passed in the HTTP header. JWT is a self-contained token that contains information about the user. JWT is signed with a secret key, so it is secure and cannot be tampered with. The body of the JWT separated by a dot (.) contains three parts: header, payload, and signature. The header contains information about the algorithm used to sign the token. The payload contains information about the user. The signature is used to verify the token.

As far as I know, many developers implement JWT in a unsafe way. They store the JWT in the client-side (e.g., local storage, session storage, or cookie). This is not safe because:

If the JWT is stolen, the attacker can impersonate the user.
If the JWT is stolen, the attacker can access until the JWT is expired.
JWT can used even if the user has logged out.

How to solve that problem? We have 2 options:

Create the token with short expiration time (e.g., 15 minutes).
Store the token to server-side (e.g., database or cache).

The first approach is good, but it has a problem. The user must re-login every 15 minutes. This is not user-friendly. The second approach is better. The token is stored in the server-side, so the user cannot access the token. The problem is server-side must do query to database check the token. This is not efficient. The best practice is to store the token in cache. In this post, I will show you how to implement JWT with Redis in Golang. I use this approach in my projects, and it works well.

Token	Expired	Function
Access Token	15 menit	User Access
Refresh Token	1 bulan	Refresh Access Token

The token is separated into 2 type: access token and refresh token. The access token is used to authenticate the user. The access token has a short expiration time (e.g., 15 minutes). The refresh token is used to get a new access token. The refresh token has a long expiration time (e.g., 7 days). The refresh token is stored in the Redis cache. When the user logs out, the refresh token is deleted from the Redis cache.

Why do we need to separate the token into 2 types?

If the access token is expired, the user must re-login. This is not user-friendly.
If the access token is stolen, the attacker can impersonate the user. The access token has a short expiration time, so the attacker cannot impersonate the user for a long time.
Logout process is easy. Just delete the refresh token from the Redis cache.

Backend

We need to define the JWT models. We have 3 models: JwtTokenDetails, JwtAuthModel, and JwtRtModel. The JwtTokenDetails model contains the access token, refresh token, access UUID, refresh UUID, access token expiration time, and refresh token expiration time. The JwtAuthModel model contains the user ID, email, first name, last name, role, and duration. The JwtRtModel model contains the user ID, email, role, and duration.

// JWT Models
package models

import "github.com/google/uuid"

type JwtTokenDetails struct {
	AccessToken  string
	RefreshToken string
	AccessUuid   string
	RefreshUuid  string
	AtExpires    int64
	RtExpires    int64
}

type JwtAuthModel struct {
	AccessId  uuid.UUID
	RefreshId uuid.UUID
	UserId    uuid.UUID
	Email     string
	FirstName string
	LastName  string
	Role      int16
	Duration  int64
}

type JwtRtModel struct {
	AccessId  uuid.UUID
	RefreshId uuid.UUID
	UserId    uuid.UUID
	Email     string
	Role      int16
	Duration  int64
}

Then we create JWT utils for handling generation of JWT. We have 2 functions: GenerateNewAuthToken and GenerateNewAccessToken. The GenerateNewAuthToken function is used to generate a new access token and refresh token. The GenerateNewAccessToken function is used to generate a new access token. The access token is signed with the secret key, and the refresh token is signed with the refresh key.

// JWT Utils
package utils

import (
	"github.com/icaksh/cripis/app/models"
	"os"
	"strconv"
	"time"

	"github.com/golang-jwt/jwt"
)

func GenerateNewAuthToken(par *models.JwtAuthModel) (*models.JwtTokenDetails, error) {
	td := &models.JwtTokenDetails{}
	var err error
	minutesCount, _ := strconv.Atoi(os.Getenv("JWT_ACCESS_TIME_KEY_EXPIRE_MINUTES_COUNT"))
	accessTime := time.Now().Add(time.Minute * time.Duration(minutesCount)).Unix()
	atm := &JWTAccess{
		AccessUuid: par.AccessId,
		User:       par.UserId,
		Role:       par.Role,
		StandardClaims: jwt.StandardClaims{
			ExpiresAt: accessTime,
		},
	}
	accessToken, err := GenerateNewAccessToken(atm)

	rtm := &JWTRefresh{
		RefreshUuid: par.RefreshId,
		User:        par.UserId,
		Email:       par.Email,
		FirstName:   par.FirstName,
		LastName:    par.LastName,
		Role:        par.Role,
		StandardClaims: jwt.StandardClaims{
			ExpiresAt: par.Duration,
		},
	}
	refreshToken, err := GenerateNewRefreshToken(rtm)

	if err != nil {
		return nil, err
	}

	td = &models.JwtTokenDetails{
		AccessUuid:   par.AccessId.String(),
		AccessToken:  accessToken,
		RefreshUuid:  par.RefreshId.String(),
		RefreshToken: refreshToken,
		AtExpires:    accessTime,
		RtExpires:    par.Duration,
	}
	return td, err
}

func GenerateNewAccessToken(atm *JWTAccess) (string, error) {
	secret := os.Getenv("JWT_SECRET_KEY")

	token := jwt.NewWithClaims(jwt.SigningMethodHS256, atm)

	accessToken, err := token.SignedString([]byte(secret))
	if err != nil {
		// Return error, it JWT token generation failed.
		return "", err
	}
	return accessToken, nil
}

func GenerateNewRefreshToken(rtm *JWTRefresh) (string, error) {
	secret := os.Getenv("JWT_REFRESH_KEY")

	token := jwt.NewWithClaims(jwt.SigningMethodHS256, rtm)

	refreshToken, err := token.SignedString([]byte(secret))
	if err != nil {
		return "nil", err
	}
	return refreshToken, nil
}

Now, we create utils for JWT validation and extraction. We have 4 functions: ExtractRefreshToken, ExtractRefreshTokenMetadata, ExtractTokenMetadata, and TokenValid. The ExtractRefreshToken function is used to extract the refresh token from the request body. The ExtractRefreshTokenMetadata function is used to extract the refresh token from the request header. The ExtractTokenMetadata function is used to extract the access token from the request header. The TokenValid function is used to check if the access token is valid.

package utils

import (
	"os"
	"strings"

	"github.com/gofiber/fiber/v2"
	"github.com/golang-jwt/jwt"
	"github.com/google/uuid"
)

type JWTAccess struct {
	AccessUuid uuid.UUID `json:"accessUuid"`
	User       uuid.UUID `json:"userId"`
	Role       int16     `json:"role"`
	jwt.StandardClaims
}

type JWTRefresh struct {
	RefreshUuid uuid.UUID `json:"refreshUuid"`
	User        uuid.UUID `json:"userId"`
	FirstName   string    `json:"firstName"`
	LastName    string    `json:"lastName"`
	Email       string    `json:"email"`
	Role        int16     `json:"role"`
	jwt.StandardClaims
}

func ExtractRefreshToken(t string) (*JWTRefresh, error) {
	token, err := jwt.ParseWithClaims(t, &JWTRefresh{}, jwtKeyFunc)
	if err != nil {
		return nil, err
	}
	claims, ok := token.Claims.(*JWTRefresh)
	if ok && token.Valid {
		return &JWTRefresh{
			RefreshUuid: claims.RefreshUuid,
			User:        claims.User,
			Email:       claims.Email,
			FirstName:   claims.FirstName,
			LastName:    claims.LastName,
			Role:        claims.Role,
			StandardClaims: jwt.StandardClaims{
				ExpiresAt: claims.ExpiresAt,
			},
		}, nil
	}
	return nil, err
}

func ExtractRefreshTokenMetadata(c *fiber.Ctx) (*JWTRefresh, error) {
	token, err := verifyRefreshToken(c)
	if err != nil {
		return nil, err
	}

	// Setting and checking token and credentials.
	claims, ok := token.Claims.(*JWTRefresh)
	if ok && token.Valid {
		// Expires time.
		return &JWTRefresh{
			RefreshUuid: claims.RefreshUuid,
			User:        claims.User,
			Email:       claims.Email,
			FirstName:   claims.FirstName,
			LastName:    claims.LastName,
			Role:        claims.Role,
			StandardClaims: jwt.StandardClaims{
				ExpiresAt: claims.ExpiresAt,
			},
		}, nil
	}

	return nil, err
}

func ExtractTokenMetadata(c *fiber.Ctx) (*JWTAccess, error) {
	token, err := verifyToken(c)
	if err != nil {
		return nil, err
	}

	claims, ok := token.Claims.(*JWTAccess)
	if ok && token.Valid {
		return &JWTAccess{
			AccessUuid: claims.AccessUuid,
			User:       claims.User,
			Role:       claims.Role,
			StandardClaims: jwt.StandardClaims{
				ExpiresAt: claims.ExpiresAt,
			},
		}, nil
	}

	return nil, err
}

func TokenValid(c *fiber.Ctx) error {
	token, err := verifyToken(c)
	if err != nil {
		return err
	}
	if _, ok := token.Claims.(jwt.Claims); !ok && !token.Valid {
		return err
	}
	return nil
}

func extractToken(c *fiber.Ctx) string {
	bearToken := c.Get("Authorization")

	// Normally Authorization HTTP header.
	onlyToken := strings.Split(bearToken, " ")
	if len(onlyToken) == 2 {
		return onlyToken[1]
	}

	return ""
}

func verifyToken(c *fiber.Ctx) (*jwt.Token, error) {
	tokenString := extractToken(c)

	token, err := jwt.ParseWithClaims(tokenString, &JWTAccess{}, jwtKeyFunc)
	if err != nil {
		return nil, err
	}

	return token, nil
}

func verifyRefreshToken(c *fiber.Ctx) (*jwt.Token, error) {
	tokenString := extractToken(c)

	token, err := jwt.ParseWithClaims(tokenString, &JWTRefresh{}, jwtKeyFunc)
	if err != nil {
		return nil, err
	}

	return token, nil
}
func jwtKeyFunc(token *jwt.Token) (interface{}, error) {
	return []byte(os.Getenv("JWT_SECRET_KEY")), nil
}
func jwtRefreshKeyFunc(token *jwt.Token) (interface{}, error) {
	return []byte(os.Getenv("JWT_REFRESH_KEY")), nil
}

Next, we need to create the JWT queries for Redis. We use the github.com/go-redis/redis package to connect to the Redis database. We have 3 functions: CreateAuth, FetchAuth and DeleteAuth. The CreateAuth function is used to create the refresh token in the Redis cache. The FetchAuth function is used to get the refresh token from the Redis cache. The DeleteAuth function is used to delete the refresh token from the Redis cache.

package queries

import (
	"fmt"
	"github.com/go-redis/redis/v7"
	"github.com/google/uuid"
	"github.com/icaksh/cripis/app/models"
	"github.com/icaksh/cripis/app/utils"
	"strconv"
	"time"
)

type JWTQueries struct {
	*redis.Client
}

func (q *JWTQueries) CreateAuth(userId uuid.UUID, td *models.JwtTokenDetails) error {
	fmt.Println(td)
	at := time.Until(time.Unix(td.AtExpires, 0))
	rt := time.Until(time.Unix(td.RtExpires, 0))
	fmt.Println(rt)
	errAccess := q.Set(td.AccessUuid, userId, at).Err()
	if errAccess != nil {
		return errAccess
	}
	errRefresh := q.Set(td.RefreshUuid, userId, rt).Err()
	if errRefresh != nil {
		return errRefresh
	}
	return nil
}

func (q *JWTQueries) FetchAuth(authD *utils.JWTAccess) (uint64, error) {
	userid, err := q.Get(authD.AccessUuid.String()).Result()
	if err != nil {
		return 0, err
	}
	userID, _ := strconv.ParseUint(userid, 10, 64)
	return userID, nil
}

func (q *JWTQueries) DeleteAuth(uuid uuid.UUID) (int64, error) {
	deleted, err := q.Del(uuid.String()).Result()
	if err != nil {
		return 0, err
	}
	return deleted, nil
}

After we create the queries, we can make database connection. We have 2 functions: RedisConnection and RedisConnect. The RedisConnection function is used to connect to the Redis database. The RedisConnect function is used to connect to the Redis database and return the JWT queries.

package database

import (
	"github.com/go-redis/redis/v7"
	"os"
)

type RedisQueries struct {
	*queries.JWTQueries
}

func RedisConnection() (*redis.Client, error) {
	client := redis.NewClient(&redis.Options{
		Username: os.Getenv("REDIS_SERVER_NAME"),
		Password: os.Getenv("REDIS_SERVER_PASS"),
		Addr:     os.Getenv("REDIS_SERVER_URL"),
		DB:       0,
	})
	_, err := client.Ping().Result()
	if err != nil {
		return nil, err
	}
	return client, nil
}

func RedisConnect() (*RedisQueries, error) {
	redis, err := RedisConnection()
	if err != nil {
		return nil, err
	}

	return &RedisQueries{
		JWTQueries: &queries.JWTQueries{Client: redis},
	}, nil

}

Now we need to create the auth controller. We have 2 functions: Login and Logout. The Login function is used to authenticate the user and generate the access token and refresh token. The Logout function is used to delete the refresh token from the Redis cache.

// Auth Controller
package controllers

import (
	"fmt"
	"github.com/google/uuid"
	"github.com/icaksh/cripis/app/models"
	"github.com/icaksh/cripis/app/utils"
	"github.com/icaksh/cripis/platform/database"
	"github.com/thanhpk/randstr"
	"os"
	"strconv"
	"strings"
	"time"

	"github.com/gofiber/fiber/v2"
	"golang.org/x/crypto/bcrypt"
)

func Login(c *fiber.Ctx) error {
	// Login Logic

	hoursCount, _ := strconv.Atoi(os.Getenv("JWT_REFRESH_TIME_KEY_EXPIRE_HOURS_COUNT"))

	if !creds.Remember {
		hoursCount, _ = strconv.Atoi(os.Getenv("JWT_REFRESH_TIME_KEY_EXPIRE_HOURS_COUNT_REMEMBER"))
	}
	refreshTime := time.Now().Add(time.Hour * time.Duration(hoursCount)).Unix()
	au := &models.JwtAuthModel{
		AccessId:  uuid.New(),
		RefreshId: uuid.New(),
		UserId:    auth.ID,
		Email:     auth.Email,
		FirstName: auth.FirstName,
		LastName:  auth.LastName,
		Role:      auth.Roles,
		Duration:  refreshTime,
	}
	token, err := utils.GenerateNewAuthToken(au)
	if err != nil {
		return utils.InternalServerError(c, err)
	}

	redisDb, err := database.RedisConnect()
	if err != nil {
		return utils.InternalServerError(c, err)
	}
	refresh := redisDb.CreateAuth(auth.ID, token)
	if refresh != nil {
		return utils.InternalServerError(c, err)
	}
	tokens := map[string]string{
		"access_token":  token.AccessToken,
		"refresh_token": token.RefreshToken,
	}
	return c.Status(fiber.StatusCreated).JSON(tokens)
}

func Logout(c *fiber.Ctx) error {

	au, err := utils.ExtractTokenMetadata(c)
	fmt.Println(au.AccessUuid)
	if err != nil {
		return c.Status(fiber.StatusUnauthorized).JSON(fiber.Map{
			"error":   true,
		})
	}
	redisDb, err := database.RedisConnect()

	if err != nil {
		return c.Status(fiber.StatusInternalServerError).JSON(fiber.Map{
			"error":   true,
		})
	}

	deleted, err := redisDb.DeleteAuth(au.AccessUuid)
	if err != nil || deleted == 0 {
		return c.Status(fiber.StatusUnauthorized).JSON(fiber.Map{
			"error":   true,
		})
	}

	return c.Status(fiber.StatusOK).JSON(fiber.Map{
		"message": "Berhasil keluar",
	})
}

We also need to create the refresh token controller. We have 1 function: RefreshToken. The RefreshToken function is used to refresh the access token. The refresh token is sent in the request body, extracted, and then the user ID is fetched from the Redis cache. After that, the access token is generated with the new expiration time.

package controllers

import (
	"fmt"
	"github.com/google/uuid"
	"github.com/icaksh/cripis/app/models"
	"github.com/icaksh/cripis/app/utils"
	"github.com/icaksh/cripis/platform/database"
	"github.com/thanhpk/randstr"
	"os"
	"strconv"
	"strings"
	"time"

	"github.com/gofiber/fiber/v2"
	"golang.org/x/crypto/bcrypt"
)

func RefreshToken(c *fiber.Ctx) error {
	body := models.RefreshToken{}
	err := c.BodyParser(&body)

	if err != nil {
		return utils.BadRequest(c, err)
	}

	validate := utils.NewValidator()
	if err := validate.Struct(&body); err != nil {
		return utils.BadRequest(c, err)
	}

	redisDb, err := database.RedisConnect()

	if err != nil {
		return utils.InternalServerError(c, err)
	}

	claims, err := utils.ExtractRefreshToken(body.RefreshToken)
	if err != nil {
		return c.Status(fiber.StatusUnauthorized).JSON(fiber.Map{
			"error":   true,
			"message": "Invalid token",
		})
	}

	deleted, err := redisDb.DeleteAuth(claims.RefreshUuid)

	if err != nil || deleted == 0 {
		return c.Status(fiber.StatusNotFound).JSON(fiber.Map{
			"error":   true,
			"message": "Invalid token",
		})
	}

	au := &models.JwtAuthModel{
		AccessId:  uuid.New(),
		RefreshId: uuid.New(),
		UserId:    claims.User,
		Email:     claims.Email,
		FirstName: claims.FirstName,
		LastName:  claims.LastName,
		Role:      claims.Role,
		Duration:  claims.ExpiresAt,
	}

	token, err := utils.GenerateNewAuthToken(au)
	if err != nil {
		return utils.InternalServerError(c, err)
	}

	refresh := redisDb.CreateAuth(claims.User, token)
	if refresh != nil {
		return utils.InternalServerError(c, err)
	}
	tokens := map[string]string{
		"access_token":  token.AccessToken,
		"refresh_token": token.RefreshToken,
	}

	return c.Status(fiber.StatusCreated).JSON(tokens)
}

We have finished building the logic we need to generate, validate, and refresh the token. Now we need to create the middleware to protect the route. We have 1 function: JWTProtected. The JWTProtected function is used to check if the access token is valid. The access token is sent in the request header, extracted, and then the user ID is fetched from the Redis cache. If the access token is valid, the user is authenticated. If the access token is invalid, the user is not authenticated.

package middleware

import (
	"github.com/gofiber/fiber/v2"
	"github.com/icaksh/cripis/app/utils"
)

func JWTProtected(c *fiber.Ctx) error {
	err := utils.TokenValid(c)
	if err != nil {
		return c.Status(fiber.StatusUnauthorized).JSON(fiber.Map{
			"error": true,
			"msg":   err.Error(),
		})
	}
	c.Next()
	return nil
}

Last, in routes, add routes to refresh token and implement middleware to protect the route.

package routes

import (
	"github.com/icaksh/cripis/app/controllers"
	"github.com/icaksh/cripis/pkg/middleware"
	recaptcha "github.com/jansvabik/fiber-recaptcha"

	"github.com/gofiber/fiber/v2"
)

func AuthRoutes(private fiber.Router, public fiber.Router) {
	pathName := "/auth"
	public.Post(pathName+"/login", recaptcha.Middleware, controllers.Login)
	public.Post(pathName+"/refresh", controllers.RefreshToken)
	public.Post(pathName+"/register", recaptcha.Middleware, controllers.Register)
	public.Post(pathName+"/reset", recaptcha.Middleware, controllers.Reset)
	//
	private.Post(pathName+"/logout", middleware.JWTProtected, controllers.Logout)
}

func UserRoutes(private fiber.Router, public fiber.Router) {
	pathName := "/user"
	private.Get(pathName+"s", middleware.JWTProtected, controllers.GetUsers)
	private.Get(pathName+"/", middleware.JWTProtected, controllers.GetUser)
	private.Get(pathName+"/:id", middleware.JWTProtected, controllers.GetUserById)
	private.Put(pathName+"/", middleware.JWTProtected, controllers.EditUser)
	private.Put(pathName+"/password", middleware.JWTProtected, controllers.EditUserPassword)
	private.Put(pathName+"/status", middleware.JWTProtected, controllers.EditUserStatus)
	private.Delete(pathName+"/:id", middleware.JWTProtected, controllers.DeleteUser)
	//
	public.Get(pathName+"/roles", controllers.GetUserRoles)
}

Frontend

Handling token in frontend is easy. Just store the token in the local storage and configure the Axios to send the token in the header using interceptors when request. Here is the example code:

axios.interceptors.request.use(
    (config) => {
        const user = JSON.parse(fastLocalStorage.getItem('user'));
        if (user && user.access_token) {
            config.headers.Authorization = `Bearer ${user.access_token}`;
        }
        return config;
    },
    (error) => {
        return Promise.reject(error);
    }
);

Now, the token is sent in the header when the request is sent. If the token is expired, the server will return a 401 status code. We need to handle this error. If the server returns a 401 status code, we need to send a request to the server to get a new access token using the refresh token and send the original request back. Here is the example code:

axios.interceptors.response.use(
    (response) => {
        return response;
    },
    async (error) => {
        const originalRequest = error.config;
        if (error.response.status === 401 && !originalRequest._retry) {
            originalRequest._retry = true;
            try {
                const user = JSON.parse(fastLocalStorage.getItem('user'));
                const uninterceptedAxiosInstance = axios.create();
                const response = await uninterceptedAxiosInstance.post(process.env.VUE_APP_BACKEND_URL + '/public/auth/refresh', {
                    refresh_token: user.refresh_token,
                });
                fastLocalStorage.removeItem('user')
                fastLocalStorage.setItem('user', JSON.stringify(response.data));
                originalRequest.headers.Authorization = `Bearer ${response.data.access_token}`;
                return axios(originalRequest);
            } catch (refreshError) {
                Vue.$store.dispatch('logout');
            }
        }
        return Promise.reject(error);
    }
);

That's it! Now you have implemented JWT with Redis in Golang. This approach is secure and efficient. The token is stored in the Redis cache, so the user cannot access the token. The refresh token is used to get a new access token. The refresh token has a long expiration time, so the user does not need to re-login frequently. The refresh token is deleted from the Redis cache when the user logs out.

[!NOTE] What if the refresh token is stolen? That is a good question. But I don't know exactly the answer. That's a risk we have to take. If the refresh token is stolen, the attacker can get a new access token. But the access token has a short expiration time, so the attacker cannot impersonate the user for a long time. The refresh token is deleted from the Redis cache when the user logs out. So, the risk is minimized.

For full implementation, you can check this repository

::github{repo="icaksh/cripis"} ::github{repo="icaksh/cripis-fe"}

// END

Cloud Run Protection with Cloudflare

Mon, 12 Feb 2024 00:00:00 GMT

I have a simple REST API deployed on Cloud Run. The application is simple, just checking daily facts from Wikipedia. Well, since I have a high open-source spirit (wkakakak), I decided to add one feature that I think is not too crucial before I publish the source code. The feature is Check the Latest Earthquake Information. Why did I choose Cloud Run? It's free up to 2 million requests per month, easy to deploy, cheap (I only pay less than 150 IDR (0.01 USD) per month for Golang apps. However, there is one problem that I encountered.

In 2023, I got a pretty big DDoS attack. Actually, it's not a DDoS, but there is a user who accesses the API with a request count per second. Unfortunately, I only realized this when the end of the month and my Cloud Run bill skyrocketed. The plot twist of the attack is that the attack accessed the Latest Earthquake API which is not too crucial.

Pointing Domain to Cloud Run

Cloud Run by default doesnt have DDoS protection. The original DDoS protection in Google Cloud is Cloud Armor. However, Cloud Armor is not free. I have to pay for the Cloud Armor service. I use Cloudflare as a free DDoS protection service, so I decided to use Cloudflare to protect my Cloud Run.

First, we need to add domain to Cloudflare.

Add domain linked on Cloud Run to Cloudflare.
In Cloudflare DNS, go to Rules > Configuration Rules and + Create Rules
Fill the configuration as follows:
- Fill the Rule Name
- Checklist Custom filter expression
- Set When incoming requests match to Hostname equals yourdomain.com
- Set Then the settings are to Automatic HTTPS Rewrites Off, Security Level High, SSL/TLS Full
- Click Deploy

Enable WAF

Remember my REST API that was accessed with a request count per second?. In the Cloud Run log, I saw that the request came from an IP owned by Amazon. Well, at first I thought that I could just block the IP. However, after checking further, it turns out that the request IP keeps changing. As a DevOps, I know that the IP will keep changing. Instead of thinking about the IP + CIDR, I just block the Amazon ASN 😁.

In Cloudflare, go to Security > WAF then + Create Rule
For example, I will block Amazon ASN. Fill the configuration as follows:
- Fill the Rule Name
- Set Expression to ASN Num equals AS16509
- Set Action to Block
- Click Deploy

Result

Now, my Cloud Run is protected by Cloudflare. I have added the domain to Cloudflare and enabled WAF to block the Amazon ASN.

Remember to turn on the Cloudflare orange cloud to enable the Cloudflare proxy. If you don't turn on the orange cloud, Cloudflare won't protect your Cloud Run.

// END

Bastille as FreeBSD Jail Management

Wed, 17 Jan 2024 00:00:00 GMT

One of the features I like in FreeBSD is jail. Jail is a feature that allows us to create a virtual environment isolated from the main operating system. I use jail because I want to keep every installed application organized so things don’t get messy.

I preferred creating jails manually. You can check my configuration at shinsha-jail. Some might ask, why do it manually? I use vtnet as the jail network interface. I find it too complicated to configure vtnet manually when the jail is activated. Moreover, I need port forwarding for each jail I set up (perhaps similar to OpenVZ?). So, I prefer the manual approach rather than using existing tools—after all, they are just as complicated.

Until, my shinsha-jail caused Google Cloud Engine (GCE) to be unable to boot when an additional bridge0 interface was added

I was actually quite skeptical about jail management tools because, the concept is the same, not much different from manual configuration. However, since I was tired of seeing my GCE serial console stuck at DHCP renewal in 18000 seconds, I decided to try using a jail management tool.

I tried ezjail, but... I didn’t really like it because I still had to configure pf manually. I have read bastille, reading container statement like docker make me sceptical. But, I decided to try it and I was surprised. Bastille is a great tool for managing jails. It is easy to use and has a lot of features.

Installing Bastille

Bastille have a good documentation for someone who first using it. You can check it at this post.

Why I Use Bastille

I use Bastille because it is easy to use and has a lot of features. I can create a jail with a single command. I can also easily manage jails, start, stop, and delete them. Bastille also has a template system that allows me to create jails with pre-configured settings.

What I love next that Bastille has a ZFS integration. I can create a jail with a ZFS dataset. This is very useful because I can easily manage the storage of the jail. I can also easily take snapshots of the jail and rollback to a previous state if something goes wrong. Bastille also has a function to easily mount a directory from the host to the jail. This is very useful because I can easily share files between the host and the jail.

I'am to lazy to configure pf manually. Bastille has a function to automatically configure pf for the jail. This is very useful because I can easily configure the firewall for the jail. I can also easily configure port forwarding for the jail.

So, let's try Bastille. I hope you find it useful. Feel free to reach out to me if you have any questions or feedback.

// END