MySQL deployment issue on IBM Cloud’s Managed Kubernetes Service due to lost+found directory
Overview
This story is about a directory (lost+found) that caused an issue in the deployment of MySQL on IBM Cloud’s Managed Kubernetes Service and how it was solved.
Details
I tried to deployed MySQL on k8s cluster using the manifest is given below:
but I got the following errors:
2019-08-25T15:36:470279Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting2019-08-09T15:36:470367Z 0 [ERROR] Aborting
After debugging the issue I found out that:
mysql container’s entrypoint script checks if DATADIR/mysql (in my case it was /var/lib/mysql) exists. If not, it run --initialize, but if DATADIR/mysql contains any files other than ones starting with . or specified with --ignore-db-dir, --initialize will fail with the above error message.The error was due to the lost+found directory that is being added in DATADIR/mysql directory when a volume is mounted and it is the default behaviour according to IBM Cloud support.
What purpose does lost+found serve?
VOBs (Versioned Object Bases) store file system objects and metadata. File system objects are files and directories. These objects are stored as elements, each of which contains one or more versions.
A file element contains one or more versions of a file. A directory element contains one or more versions of a directory, each of which can contain file elements and other directory elements.
Versions of text files are distinguished by changes in their text. Versions of directories are distinguished by changes in their contents (to account for files and subdirectories that have been renamed, removed, or added).
Every VOB includes a special directory element, lost+found, which is used to hold elements that become stranded when they are not catalogued in any directory version in the VOB.
An element can become stranded when you do any of the following:
- Create new elements, and then cancel the checkout of the directory in which they were created
- Delete the last reference to an element by using the rmname command
- Delete the last reference to an element by deleting a directory version with the
rmver,rmbranch, orrmelemcommand
Solution
The above issue was resolved by using an init-container. Init container’s manifest is given below:
Init container removes the lost+found directory before the MySQL server start inside the container.
Final manifest is given below:
MySQL Data Backup
At this point, MySQL will be deployed on k8s cluster. The next task will be to back up its data.
To backup MySQL data on AWS S3 bucket follow the guidelines provided in this story.
Final Thoughts
It is one of the ways that I found out to resolve this issue. If you have a better idea do let me know! :)
