Verify my understanding of migration from v2 to v3 is sane #111

FreydNot · 2023-08-31T21:58:13Z

FreydNot
Aug 31, 2023

I have an instance currently running version 2. I'd like to upgrade to version 3. I'd just start over with the pre-built image, but I'd like to preserve my historical data. So, upgrade in place it is.

If I'm understanding correctly, I just follow the instructions on the beta release page. I assume I should use "git checkout v0.3.1", right?

Sounds simple, but I have some questions.

Should I do an apt update and apt upgrade before I start? What happens if that throws errors?

Does the database undergo migration of any kind upon first run of v3? I understand there have been changes and now there is a short term, high resolution data base and a lower resolution (5 minute) long term database. Do I need to intervene in any way?

Once I have v3 running, I'm inclined to make a backup, wipe the drive, start over from the pre-built image and then restore the backup. The idea being I would be as close to a normal install as possible. Am I over thinking this?

If anyone is interested, this upgrade was prompted because I ran into issues trying to install rpi-power-monitor-mqtt. Specifically this:

pi@rpipowermonitor:~ $ sudo apt-get install influxdb-client mosquitto-clients
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
mosquitto-clients is already the newest version (2.0.11-1).
The following NEW packages will be installed:
  influxdb-client
0 upgraded, 1 newly installed, 0 to remove and 107 not upgraded.
Need to get 0 B/2,018 kB of archives.
After this operation, 6,547 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
(Reading database ... 55659 files and directories currently installed.)
Preparing to unpack .../influxdb-client_1.6.7~rc0-1_armhf.deb ...
Unpacking influxdb-client (1.6.7~rc0-1) ...
dpkg: error processing archive /var/cache/apt/archives/influxdb-client_1.6.7~rc0-1_armhf.deb (--unpack):
 trying to overwrite '/usr/bin/influx', which is also in package influxdb 1.8.10-1
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/influxdb-client_1.6.7~rc0-1_armhf.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

David00 · 2023-09-01T00:15:26Z

David00
Sep 1, 2023
Maintainer

Hey @FreydNot!

Thanks for opening an issue. That's good to know about the mqtt issue - I'm not too sure what the conflict is without diving further. But, since there's an update for my project available, that's a good place to start! You may also be interested in the MQTT plugin that @jmadden91 authored (just waiting on the final usage documentation from him)

Regarding your questions:

I assume I should use "git checkout v0.3.1", right?

Yes, that's right. Thank you for letting me know - I'll update the instructions. (Technically, v0.3.1 does not include any changes to the power monitor code - it's just an update to the custom OS image, but you can of course still checkout v0.3.1).

Should I do an apt update and apt upgrade before I start? What happens if that throws errors?

Yes, that is fine if you do it before, but you don't need to do it at all if you don't want to. The apt update and apt upgrade is going to update your underlying operating system packages, which isn't required to run v0.3.0. However, it is usually a good idea to keep your system up to date. I would not expect the update to throw any errors, but if it does, feel free to share them here and we can look at it together.

Does the database undergo migration of any kind upon first run of v3? I understand there have been changes and now there is a short term, high resolution data base and a lower resolution (5 minute) long term database. Do I need to intervene in any way?

No. The database structure is the same in v0.3.0 as v0.2.0. The database changes were really only when going from v0.1.0 to v0.2.0. You don't need to do anything to the database when upgrading from v0.2.0 to v0.3.0. For more information on the database structure and retention policies, see this page here in the new docs:
https://david00.github.io/rpi-power-monitor/docs/v0.3.0/database-info.html

Once I have v3 running, I'm inclined to make a backup, wipe the drive, start over from the pre-built image and then restore the backup. The idea being I would be as close to a normal install as possible. Am I over thinking this?

That's not an awful idea, but it also isn't necessary. On the other hand, it would be a good opportunity to replace your microSD card (if it's not a high endurance card), or upgrade to a NVMe drive. I don't currently have data restoration procedures written/published, but it's pretty straightforward with just a few CLI commands. Regardless of whether or not you decide to wipe and start fresh, you'll have to go through the new calibration setup, which thankfully is much easier and faster now. So, you'll essentially be setting up the power monitor again either way, and can carry your old data forward either way.

After you're on v0.3.0, I highly recommend enabling automatic backups for peace of mind!

Feel free to reach out here if you get stuck along the way!

0 replies

FreydNot · 2023-09-01T01:19:09Z

FreydNot
Sep 1, 2023
Author

Thanks for the quick response. If there is no database change, can I backup my current database (with your v3.0 script or by hand in a shell), start over from the prebuilt image and restore my v2 backup onto the new v3 system? That seems like the cleanest and easiest way to go.

Are the CLI commands to restore a backup documented anywhere?

1 reply

David00 Sep 1, 2023
Maintainer

Yes, you can backup your power monitor data from v0.2.0 using the backup script, or by hand, using the commands that the script runs. I've provided the manual backup commands below.

If you want to use the script, it needs to reference config.toml (click to download). See the steps here for setting up config.toml. Note that the script requires a USB device to be connected to the Pi in order to run.

Manual Backup Commands

If you've changed your database name on your current Pi, make sure to change the value after export database_name= below. Otherwise, just copy and paste both sections (line by line, as the influxd backup command will take a little while if you have a lot of data)

export database_name=power_monitor

cd ~/
mkdir temp_backup_dir
influxd backup -portable -database $database_name ./temp_backup_dir
tar -czvf ~/backup.tar.gz  temp_backup_dir

You'll be left with a single file named backup.tar.gz at your home directory, which you'll then have to transport off your Pi. The easiest way to get it off is to move it to the web folder, and then navigate to your Pi's IP in a web browser from your computer, and click the filename:

mv ~/backup.tar.gz /var/www/html/

Importing the backup

Once you're up and running on your new image - and you've transferred the backup.tar.gz file to your new environment - you can extract import the InfluxDB backup with the following commands, assuming you put the backup file into /home/pi/:

tar -xzf backup.tar.gz 
influxd restore -portable temp_backup_dir

That should be all there is to it! Note that it is a good practice to test the entire process out from start to finish before wiping the original data. You can do this on your current Pi - before wiping it - by importing the backup file into a new database with the following command:

influxd restore -portable temp_backup_dir -newdb import_test

Or, you could flash the v0.3.1 image to a new microSD card or USB drive so that you still have your original microSD card intact, just in case.

FreydNot · 2023-09-01T02:53:03Z

FreydNot
Sep 1, 2023
Author

tl;dr I tried going to automatic script route, but it didn't work. I'll move to the manual route.

I'm pretty sure I've got everything in the right place and set up properly.

pi@rpipowermonitor:~/rpi_power_monitor/rpi_power_monitor $ ls -l
total 164
-rwxr-xr-x 1 pi pi 12674 Sep  1 02:31 backup.py
-rw-r--r-- 1 pi pi  1664 Jan 27  2023 config.py
-rw-r--r-- 1 pi pi  1664 Jan 27  2023 config.py.backup
-rw-r--r-- 1 pi pi  2515 Sep  1 03:38 config.toml
drwxr-xr-x 3 pi pi  4096 Jan 27  2023 data
-rw-r--r-- 1 pi pi  8669 Nov 12  2022 influx_interface.py
-rw-r--r-- 1 pi pi    93 Nov 12  2022 __init__.py
-rw-r--r-- 1 pi pi  3688 Nov 12  2022 plotting.py
-rw-r--r-- 1 pi pi 53701 Feb  4  2023 power_monitor.py
-rw-r--r-- 1 pi pi 53737 Jan 29  2023 power_monitor.py.backup

pi@rpipowermonitor:~/rpi_power_monitor/rpi_power_monitor $ tail -5 config.toml
[backups]
backup_device = '/dev/sdb1'             # This should be the device path as shown by the command sudo fdisk -l | grep "/dev/sd*"
folder_name = 'power_monitor_backups'   # All backups will be placed into this folder at the root of your USB drive.
mount_path = '/media/backups'           # The directory where the USB drive will be mounted to. No need to change this unless you're already using this path for something else.
backup_count = 2                        # Number of backups to keep (the backup utility will automatically delete the oldest backups once the total count exceeds this number).

My system is a bit unusual. My main drive is a SATA SSD hanging off a USB adapter. This shows up as sda. My USB drive for backups shows up on sdb.

$ sudo lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0 59.6G  0 disk
├─sda1   8:1    0  256M  0 part /boot
└─sda2   8:2    0 59.4G  0 part /
sdb      8:16   1 28.9G  0 disk
└─sdb1   8:17   1 28.9G  0 part

First I tried to run the script normally.

pi@rpipowermonitor:~/rpi_power_monitor/rpi_power_monitor $ ./backup.py
./backup.py: line 7: import: command not found
./backup.py: line 8: import: command not found
./backup.py: line 9: from: command not found
./backup.py: line 10: from: command not found
./backup.py: line 11: import: command not found
./backup.py: line 12: import: command not found
./backup.py: line 13: import: command not found
./backup.py: line 14: import: command not found
./backup.py: line 15: import: command not found
./backup.py: line 16: import: command not found
./backup.py: line 17: from: command not found
./backup.py: line 18: import: command not found
./backup.py: line 21: syntax error near unexpected token `('
./backup.py: line 21: `formatter = logging.Formatter(fmt='{asctime} - {levelname}: {message}', datefmt='%m-%d-%Y %H:%M:%S', style='{')'

That's weird. Maybe I need to give it super user permissions...

pi@rpipowermonitor:~/rpi_power_monitor/rpi_power_monitor $ sudo ./backup.py
./backup.py: 7: import: not found
./backup.py: 8: import: not found
./backup.py: 9: from: not found
./backup.py: 10: from: not found
./backup.py: 11: import: not found
./backup.py: 12: import: not found
./backup.py: 13: import: not found
./backup.py: 14: import: not found
./backup.py: 15: import: not found
./backup.py: 16: import: not found
./backup.py: 17: from: not found
./backup.py: 18: import: not found
./backup.py: 21: Syntax error: "(" unexpected

¯\_(ツ)_/¯

Probably a good idea to wipe this whole thing and start over from scratch.

I'll just do it manually.

0 replies

FreydNot · 2023-09-01T06:20:06Z

FreydNot
Sep 1, 2023
Author

Well, the manual backup was successful (I think) but the test restore fails.

20230901T054616Z.manifest     20230901T054616Z.s141.tar.gz  20230901T054616Z.s197.tar.gz  20230901T054616Z.s237.tar.gz  20230901T054616Z.s61.tar.gz
20230901T054616Z.meta         20230901T054616Z.s149.tar.gz  20230901T054616Z.s1.tar.gz    20230901T054616Z.s245.tar.gz  20230901T054616Z.s69.tar.gz
20230901T054616Z.s101.tar.gz  20230901T054616Z.s157.tar.gz  20230901T054616Z.s205.tar.gz  20230901T054616Z.s29.tar.gz   20230901T054616Z.s77.tar.gz
20230901T054616Z.s117.tar.gz  20230901T054616Z.s165.tar.gz  20230901T054616Z.s213.tar.gz  20230901T054616Z.s37.tar.gz   20230901T054616Z.s85.tar.gz
20230901T054616Z.s125.tar.gz  20230901T054616Z.s173.tar.gz  20230901T054616Z.s21.tar.gz   20230901T054616Z.s45.tar.gz   20230901T054616Z.s93.tar.gz
20230901T054616Z.s133.tar.gz  20230901T054616Z.s181.tar.gz  20230901T054616Z.s221.tar.gz  20230901T054616Z.s53.tar.gz
20230901T054616Z.s13.tar.gz   20230901T054616Z.s189.tar.gz  20230901T054616Z.s229.tar.gz  20230901T054616Z.s5.tar.gz

pi@rpipowermonitor:~ $ sudo influxd restore -portable temp_backup_dir -newdb import_test
2023/09/01 07:11:01 error updating meta: DB metadata not changed. database may already exist
restore: DB metadata not changed. database may already exist

I guess it's possible I've done this before. I'll try a different database name...

pi@rpipowermonitor:~ $ sudo influxd restore -portable temp_backup_dir -newdb xyzzy
2023/09/01 07:12:57 error updating meta: DB metadata not changed. database may already exist
restore: DB metadata not changed. database may already exist

Nope. Must be some other problem.

I'm going to pause here since I can't guarantee the validity of my backup.

1 reply

David00 Sep 1, 2023
Maintainer

The issue you had with running the backup script is because bash tried executing it instead of Python. I don't have a shebang at the top of the file to tell bash to call Python. If you try again with python3 backup.py, it should work.

I just looked into the DB metadata not changed issue, and it's because of the order in which I put the flags in the restoration command. Also, it looks like -db needs to be provided when -newdb is used, so the new import command should be:

influxd restore -portable -db power_monitor -newdb import_test temp_backup_dir/

FreydNot · 2023-09-01T19:31:08Z

FreydNot
Sep 1, 2023
Author

The updated restore command appears to have worked.

I rebooted the pi and adjusted the data source in Grafana to point to the newly created import_test database.

That also seems to have worked, but there was very little actual data restored.

There is nothing in the last 7 days:

and barely anything over the last 1 year:

Clearly I'm not ready to wipe this drive yet. I'll see if running the automatic script properly will do something different.

3 replies

David00 Sep 1, 2023
Maintainer

For this issue, you will have to edit each panel and change the data source. Here is an example:

EDIT: Or, better yet, duplicate the dashboard, and make the changes on the copy so that you can easily compare.

FreydNot Sep 1, 2023
Author

I see what you are getting at, but I don't see any place to change power_monitor to import_test (nor import_test2)

and the query inspector shows the database is import_test

David00 Sep 1, 2023
Maintainer

Ok, we'll have to dip into the command line Influx to validate. In two separate terminals, let's open a database connection to each:

window 1: influx -database power_monitor -precision rfc3339
window 2: influx -database import_test2 -precision rfc3339

Then, in each window, you can run the following commands and compare the output between the two databases:

show measurements;
select sum(power) raw_cts where time >= now() - 1d and time < now() - 20h;        ## NOTE you might have to adjust the time depending on when you took the backups. This query will look between 20 and 24 hours ago.
select sum(current) from home_load where time >= now() - 1d and time < now() - 20h;

You can do a few other spot checks by repeating the idea above for various measurements shown in the show measurement command and by adjusting the time frames, etc.

Also note that to look at "old" data - which is anything older than 30 days ago -you have to query the data from the measurement that is associated to the retention policy. This is probably why there was no old data when you went back in Grafana. Such a query would look like this:

select sum(current) from rp_5min.home_load_5m where time >= now() - 201d and time < now() - 200d;

Note the rp_5min. before, and the _5m appended to the measurement name. The RP names and associated measurement names are listed in the docs in the table in this section here:
https://david00.github.io/rpi-power-monitor/docs/v0.3.0/database-info.html#structure

I think you'll find that the backup and restoration is fine - I'm sure it has just been a series of complications disguising it otherwise :)

FreydNot · 2023-09-01T19:36:26Z

FreydNot
Sep 1, 2023
Author

Running backup.py properly still hit a snag.

pi@rpipowermonitor:~/rpi_power_monitor/rpi_power_monitor $ python3 backup.py
Traceback (most recent call last):
  File "/home/pi/rpi_power_monitor/rpi_power_monitor/backup.py", line 11, in <module>
    import tomli
ModuleNotFoundError: No module named 'tomli'
pi@rpipowermonitor:~/rpi_power_monitor/rpi_power_monitor $ sudo python3 backup.py
Traceback (most recent call last):
  File "/home/pi/rpi_power_monitor/rpi_power_monitor/backup.py", line 11, in <module>
    import tomli
ModuleNotFoundError: No module named 'tomli'

I'm going to repeat the manual backup and manual restore process again using a different new database name and take better notes on the restore process in case I missed something.

2 replies

David00 Sep 1, 2023
Maintainer

This happens because you'd need to pip install tomli - I forgot that this is a dependency for the new v0.3.0 environment (and backup script).

However, the manual commands that I shared above are essentially identical to what the script does, minus the need for an external USB.

FreydNot Sep 1, 2023
Author

That makes sense. Since the manual method seems to have worked, I'm going to stop trying the automatic script method.

FreydNot · 2023-09-01T20:02:53Z

FreydNot
Sep 1, 2023
Author

Okay, new backup attempt.

pi@rpipowermonitor:~ $ export database_name=power_monitor
pi@rpipowermonitor:~ $ cd ~/
pi@rpipowermonitor:~ $ ls -l temp_backup_dir
ls: cannot access 'temp_backup_dir': No such file or directory
pi@rpipowermonitor:~ $ mkdir temp_backup_dir
pi@rpipowermonitor:~ $ influxd backup -portable -database $database_name ./temp_backup_dir
2023/09/01 20:40:09 backing up metastore to temp_backup_dir/meta.00
2023/09/01 20:40:09 backing up db=power_monitor
2023/09/01 20:40:09 backing up db=power_monitor rp=autogen shard=1 to temp_backup_dir/power_monitor.autogen.00001.00 since 0001-01-01T00:00:00Z
2023/09/01 20:40:13 backing up db=power_monitor rp=autogen shard=5 to temp_backup_dir/power_monitor.autogen.00005.00 since 0001-01-01T00:00:00Z
2023/09/01 20:40:29 backing up db=power_monitor rp=autogen shard=13 to temp_backup_dir/power_monitor.autogen.00013.00 since 0001-01-01T00:00:00Z
2023/09/01 20:40:43 backing up db=power_monitor rp=autogen shard=21 to temp_backup_dir/power_monitor.autogen.00021.00 since 0001-01-01T00:00:00Z
2023/09/01 20:40:57 backing up db=power_monitor rp=autogen shard=29 to temp_backup_dir/power_monitor.autogen.00029.00 since 0001-01-01T00:00:00Z
2023/09/01 20:41:10 backing up db=power_monitor rp=autogen shard=37 to temp_backup_dir/power_monitor.autogen.00037.00 since 0001-01-01T00:00:00Z
2023/09/01 20:41:27 backing up db=power_monitor rp=autogen shard=45 to temp_backup_dir/power_monitor.autogen.00045.00 since 0001-01-01T00:00:00Z
2023/09/01 20:41:41 backing up db=power_monitor rp=autogen shard=53 to temp_backup_dir/power_monitor.autogen.00053.00 since 0001-01-01T00:00:00Z
2023/09/01 20:41:57 backing up db=power_monitor rp=autogen shard=61 to temp_backup_dir/power_monitor.autogen.00061.00 since 0001-01-01T00:00:00Z
2023/09/01 20:42:11 backing up db=power_monitor rp=autogen shard=69 to temp_backup_dir/power_monitor.autogen.00069.00 since 0001-01-01T00:00:00Z
2023/09/01 20:42:24 backing up db=power_monitor rp=autogen shard=77 to temp_backup_dir/power_monitor.autogen.00077.00 since 0001-01-01T00:00:00Z
2023/09/01 20:42:25 backing up db=power_monitor rp=autogen shard=93 to temp_backup_dir/power_monitor.autogen.00093.00 since 0001-01-01T00:00:00Z
2023/09/01 20:42:25 backing up db=power_monitor rp=autogen shard=101 to temp_backup_dir/power_monitor.autogen.00101.00 since 0001-01-01T00:00:00Z
2023/09/01 20:42:38 backing up db=power_monitor rp=autogen shard=109 to temp_backup_dir/power_monitor.autogen.00109.00 since 0001-01-01T00:00:00Z
2023/09/01 20:42:39 backing up db=power_monitor rp=autogen shard=117 to temp_backup_dir/power_monitor.autogen.00117.00 since 0001-01-01T00:00:00Z
2023/09/01 20:42:52 backing up db=power_monitor rp=autogen shard=125 to temp_backup_dir/power_monitor.autogen.00125.00 since 0001-01-01T00:00:00Z
2023/09/01 20:43:04 backing up db=power_monitor rp=autogen shard=133 to temp_backup_dir/power_monitor.autogen.00133.00 since 0001-01-01T00:00:00Z
2023/09/01 20:43:18 backing up db=power_monitor rp=autogen shard=141 to temp_backup_dir/power_monitor.autogen.00141.00 since 0001-01-01T00:00:00Z
2023/09/01 20:43:31 backing up db=power_monitor rp=autogen shard=149 to temp_backup_dir/power_monitor.autogen.00149.00 since 0001-01-01T00:00:00Z
2023/09/01 20:43:46 backing up db=power_monitor rp=autogen shard=157 to temp_backup_dir/power_monitor.autogen.00157.00 since 0001-01-01T00:00:00Z
2023/09/01 20:44:00 backing up db=power_monitor rp=autogen shard=165 to temp_backup_dir/power_monitor.autogen.00165.00 since 0001-01-01T00:00:00Z
2023/09/01 20:44:14 backing up db=power_monitor rp=autogen shard=173 to temp_backup_dir/power_monitor.autogen.00173.00 since 0001-01-01T00:00:00Z
2023/09/01 20:44:28 backing up db=power_monitor rp=autogen shard=181 to temp_backup_dir/power_monitor.autogen.00181.00 since 0001-01-01T00:00:00Z
2023/09/01 20:44:42 backing up db=power_monitor rp=autogen shard=189 to temp_backup_dir/power_monitor.autogen.00189.00 since 0001-01-01T00:00:00Z
2023/09/01 20:44:55 backing up db=power_monitor rp=autogen shard=197 to temp_backup_dir/power_monitor.autogen.00197.00 since 0001-01-01T00:00:00Z
2023/09/01 20:45:08 backing up db=power_monitor rp=autogen shard=205 to temp_backup_dir/power_monitor.autogen.00205.00 since 0001-01-01T00:00:00Z
2023/09/01 20:45:23 backing up db=power_monitor rp=autogen shard=213 to temp_backup_dir/power_monitor.autogen.00213.00 since 0001-01-01T00:00:00Z
2023/09/01 20:45:36 backing up db=power_monitor rp=autogen shard=221 to temp_backup_dir/power_monitor.autogen.00221.00 since 0001-01-01T00:00:00Z
2023/09/01 20:45:49 backing up db=power_monitor rp=autogen shard=229 to temp_backup_dir/power_monitor.autogen.00229.00 since 0001-01-01T00:00:00Z
2023/09/01 20:46:00 backing up db=power_monitor rp=autogen shard=237 to temp_backup_dir/power_monitor.autogen.00237.00 since 0001-01-01T00:00:00Z
2023/09/01 20:46:13 backing up db=power_monitor rp=autogen shard=245 to temp_backup_dir/power_monitor.autogen.00245.00 since 0001-01-01T00:00:00Z
2023/09/01 20:46:24 backup complete:
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.meta
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s1.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s5.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s13.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s21.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s29.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s37.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s45.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s53.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s61.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s69.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s77.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s93.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s101.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s109.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s117.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s125.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s133.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s141.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s149.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s157.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s165.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s173.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s181.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s189.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s197.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s205.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s213.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s221.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s229.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s237.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.s245.tar.gz
2023/09/01 20:46:24     temp_backup_dir/20230901T194009Z.manifest

Doesn't look it it hit any errors.

pi@rpipowermonitor:~ $ du -h
48K     ./rpi_power_monitor/rpi_power_monitor/data/samples
52K     ./rpi_power_monitor/rpi_power_monitor/data
216K    ./rpi_power_monitor/rpi_power_monitor
28K     ./rpi_power_monitor/rpi_power_monitor.egg-info
8.0K    ./rpi_power_monitor/.git/info
4.0K    ./rpi_power_monitor/.git/objects/info
40K     ./rpi_power_monitor/.git/objects/pack
48K     ./rpi_power_monitor/.git/objects
4.0K    ./rpi_power_monitor/.git/branches
8.0K    ./rpi_power_monitor/.git/logs
64K     ./rpi_power_monitor/.git/hooks
4.0K    ./rpi_power_monitor/.git/refs/heads
4.0K    ./rpi_power_monitor/.git/refs/tags
12K     ./rpi_power_monitor/.git/refs
172K    ./rpi_power_monitor/.git
84K     ./rpi_power_monitor/build/lib/rpi_power_monitor
88K     ./rpi_power_monitor/build/lib
4.0K    ./rpi_power_monitor/build/bdist.linux-armv7l
96K     ./rpi_power_monitor/build
568K    ./rpi_power_monitor
8.0K    ./.ssh
1.8G    ./temp_backup_dir
1.9G    ./old_backup_dir
32K     ./.cache/pip/http/a/1/9/5/3
36K     ./.cache/pip/http/a/1/9/5
40K     ./.cache/pip/http/a/1/9
44K     ./.cache/pip/http/a/1
48K     ./.cache/pip/http/a
12K     ./.cache/pip/http/d/d/d/1/8
16K     ./.cache/pip/http/d/d/d/1
20K     ./.cache/pip/http/d/d/d
24K     ./.cache/pip/http/d/d
28K     ./.cache/pip/http/d
80K     ./.cache/pip/http
8.0K    ./.cache/pip/selfcheck
92K     ./.cache/pip
96K     ./.cache
7.4G    .

Looks like the new backup is slightly smaller than my pervious attempt. Both are fairly large at 1.8ish Gb. Seems reasonable.

I'll skip the tar creation since I'm just going to restore the backup straitaway.

pi@rpipowermonitor:~ $ influxd restore -portable -db power_monitor -newdb import_test2 temp_backup_dir/
2023/09/01 20:55:51 Restoring shard 205 live from backup 20230901T194009Z.s205.tar.gz
2023/09/01 20:55:55 Restoring shard 221 live from backup 20230901T194009Z.s221.tar.gz
2023/09/01 20:56:02 Restoring shard 29 live from backup 20230901T194009Z.s29.tar.gz
2023/09/01 20:56:07 Restoring shard 125 live from backup 20230901T194009Z.s125.tar.gz
2023/09/01 20:56:13 Restoring shard 133 live from backup 20230901T194009Z.s133.tar.gz
2023/09/01 20:56:19 Restoring shard 173 live from backup 20230901T194009Z.s173.tar.gz
2023/09/01 20:56:24 Restoring shard 189 live from backup 20230901T194009Z.s189.tar.gz
2023/09/01 20:56:29 Restoring shard 69 live from backup 20230901T194009Z.s69.tar.gz
2023/09/01 20:56:33 Restoring shard 37 live from backup 20230901T194009Z.s37.tar.gz
2023/09/01 20:56:39 Restoring shard 53 live from backup 20230901T194009Z.s53.tar.gz
2023/09/01 20:56:44 Restoring shard 77 live from backup 20230901T194009Z.s77.tar.gz
2023/09/01 20:56:44 Restoring shard 213 live from backup 20230901T194009Z.s213.tar.gz
2023/09/01 20:56:50 Restoring shard 229 live from backup 20230901T194009Z.s229.tar.gz
2023/09/01 20:56:56 Restoring shard 1 live from backup 20230901T194009Z.s1.tar.gz
2023/09/01 20:56:58 Restoring shard 93 live from backup 20230901T194009Z.s93.tar.gz
2023/09/01 20:56:58 Restoring shard 109 live from backup 20230901T194009Z.s109.tar.gz
2023/09/01 20:56:58 Restoring shard 141 live from backup 20230901T194009Z.s141.tar.gz
2023/09/01 20:57:04 Restoring shard 165 live from backup 20230901T194009Z.s165.tar.gz
2023/09/01 20:57:08 Restoring shard 13 live from backup 20230901T194009Z.s13.tar.gz
2023/09/01 20:57:15 Restoring shard 21 live from backup 20230901T194009Z.s21.tar.gz
2023/09/01 20:57:20 Restoring shard 45 live from backup 20230901T194009Z.s45.tar.gz
2023/09/01 20:57:24 Restoring shard 117 live from backup 20230901T194009Z.s117.tar.gz
2023/09/01 20:57:30 Restoring shard 245 live from backup 20230901T194009Z.s245.tar.gz
2023/09/01 20:57:34 Restoring shard 5 live from backup 20230901T194009Z.s5.tar.gz
2023/09/01 20:57:37 Restoring shard 149 live from backup 20230901T194009Z.s149.tar.gz
2023/09/01 20:57:41 Restoring shard 237 live from backup 20230901T194009Z.s237.tar.gz
2023/09/01 20:57:47 Restoring shard 61 live from backup 20230901T194009Z.s61.tar.gz
2023/09/01 20:57:51 Restoring shard 101 live from backup 20230901T194009Z.s101.tar.gz
2023/09/01 20:57:57 Restoring shard 157 live from backup 20230901T194009Z.s157.tar.gz
2023/09/01 20:58:02 Restoring shard 181 live from backup 20230901T194009Z.s181.tar.gz
2023/09/01 20:58:08 Restoring shard 197 live from backup 20230901T194009Z.s197.tar.gz

Again, no errors. Looks reasonably successful.

Now Change the database source in Grafana.

My first though is to reboot, but now I'm wondering if there is some process going on in the background that I don't want to interrupt.

I'll stop here and give it some time.

0 replies

FreydNot · 2023-09-02T01:52:36Z

FreydNot
Sep 2, 2023
Author

So, I decided it would be easiest to just spin up a second instance of power-monitor on a second Rpi and a second hard drive. Everything went smoothly with the data restore and now my restored Grafana dashboards match the current one.

Problem solved. Moving along to reinstalling.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Verify my understanding of migration from v2 to v3 is sane #111

{{title}}

Replies: 8 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Verify my understanding of migration from v2 to v3 is sane #111

FreydNot Aug 31, 2023

Replies: 8 comments · 7 replies

David00 Sep 1, 2023 Maintainer

FreydNot Sep 1, 2023 Author

David00 Sep 1, 2023 Maintainer

Manual Backup Commands

Importing the backup

FreydNot Sep 1, 2023 Author

FreydNot Sep 1, 2023 Author

David00 Sep 1, 2023 Maintainer

FreydNot Sep 1, 2023 Author

David00 Sep 1, 2023 Maintainer

FreydNot Sep 1, 2023 Author

David00 Sep 1, 2023 Maintainer

FreydNot Sep 1, 2023 Author

David00 Sep 1, 2023 Maintainer

FreydNot Sep 1, 2023 Author

FreydNot Sep 1, 2023 Author

FreydNot Sep 2, 2023 Author

FreydNot
Aug 31, 2023

Replies: 8 comments 7 replies

David00
Sep 1, 2023
Maintainer

FreydNot
Sep 1, 2023
Author

David00 Sep 1, 2023
Maintainer

FreydNot
Sep 1, 2023
Author

FreydNot
Sep 1, 2023
Author

David00 Sep 1, 2023
Maintainer

FreydNot
Sep 1, 2023
Author

David00 Sep 1, 2023
Maintainer

FreydNot Sep 1, 2023
Author

David00 Sep 1, 2023
Maintainer

FreydNot
Sep 1, 2023
Author

David00 Sep 1, 2023
Maintainer

FreydNot Sep 1, 2023
Author

FreydNot
Sep 1, 2023
Author

FreydNot
Sep 2, 2023
Author