troubleshooting database backups from OCI Console

 In this blog, we are going to check how we can troubleshoot OCI VM DB system Database console backups. OCI VM DB system is a PaaS product offering from Oracle and thus if we encounter any errors we will have to raise SR with Oracle and provide them the logs. But before going to Oracle support for OCI VM DB System backup issues, there are some checks which we can also do it from our side and can also resolve the issue in a much faster time frame. 

Well in cloud, everything is assumed to be done on click click and click. Things aren't so easy as it appears in the console. Underlying the console, there are various locations which we should be aware of for troubleshooting any stuff in OCI. I will go the console and show you first on how we can take the manual backup for OCI VM DB System.

Now, depending upon your DB Size, the backup will take time. Ultimately, once the backup process ends, you will see the below screen which will tell you whether the backup is completed or failed.

This is place where the actually help is required and troubleshooting will begin.

We will login to the DB server using the root user.

[root@reportingdb opc]# /opt/oracle/dcs/bin/dbcli list-databases ID DB Name DB Type DB Version CDB Class Shape Storage Status DbHomeID ---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ---------------------------------------- c2ea8791-d9c5-4ef5-abfe-a2bbf2b5cc6d TEST Si 19.6.0.0.0 true Oltp ASM Configured 6a48bd01-cb34-4734-b070-939db31b49df [root@reportingdb opc]#

Now we, will list down the jobs which had run against the database

[root@TEST opc]# /opt/oracle/dcs/bin/dbcli list-jobs|grep TEST|tail -10 9f50994c-1c6e-4aa6-9c47-25fa991a8b23 Remove Audit files for DB: TEST August 22, 2021 12:00:00 AM UTC Success f4930fbd-93a3-45bd-89b9-48428385f94c Remove Audit files for DB: TEST August 23, 2021 12:00:00 AM UTC Success 2fa4c328-4246-4a2d-b407-0fdc2a353668 Remove Audit files for DB: TEST August 24, 2021 12:00:00 AM UTC Success 628c9970-dcc3-40e6-b355-cb903715611b Remove Audit files for DB: TEST August 25, 2021 12:00:00 AM UTC Success 3a082250-47e7-4bb0-ae58-8ed34c44c55f Remove Audit files for DB: TEST August 26, 2021 12:00:00 AM UTC Success 565bea8e-ca5f-48de-9110-d4c99ca98e02 Remove Audit files for DB: TEST August 27, 2021 12:00:00 AM UTC Success 0ff5a34c-36ce-4b10-9632-48843b21fd53 Remove Audit files for DB: TEST August 28, 2021 12:00:00 AM UTC Success dbb49b4e-0035-46a7-9a74-dd493de8c264 Remove Audit files for DB: TEST August 29, 2021 12:00:00 AM UTC Success 2f77b738-be5f-4986-b264-34a619f42159 Remove Audit files for DB: TEST August 30, 2021 12:00:00 AM UTC Success 01edcd9a-553d-4117-9e98-e92b8c2aa23e update database : TEST August 30, 2021 8:57:04 AM UTC Failure [root@reportingdb opc]#

Let us check the failed one

[root@TEST opc]# /opt/oracle/dcs/bin/dbcli describe-job -i 01edcd9a-553d-4117-9e98-e92b8c2aa23e -j Error reading entity from input stream. [root@reportingdb opc]#

Now we don't get much information from here, so it is worth checking the dcs agent log

location:-/opt/oracle/dcs/log

{ "updatedTime" : "Mon Aug 30, 2021 08:57:23.072 (UTC) [1630313843072]", "startTime" : "Mon Aug 30, 2021 08:57:20.269 (UTC) [1630313840269]", "endTime" : "Mon Aug 30, 2021 08:57:23.069 (UTC) [1630313843069]", "taskId" : "TaskZJsonRpcExt_1860", "status" : "Failure", "taskResult" : "DCS-10045:Validation error encountered: OMF parameters.OMF parameter settings are not as per DbStorage Location ", "taskName" : "Validate OMF parameter values", "taskDescription" : null, "parentTaskId" : "TaskSequential_1843", "jobId" : "01edcd9a-553d-4117-9e98-e92b8c2aa23e", "message" : null, "tags" : [ ], "reportLevel" : "Error" } ], "createTimestamp" : "Mon Aug 30, 2021 08:57:04.662 (UTC) [1630313824662]", "percentageProgress" : "4%", "resourceList" : [ { "updatedTime" : "Mon Aug 30, 2021 08:57:04.679 (UTC) [1630313824679]", "resourceId" : "c2ea8791-d9c5-4ef5-abfe-a2bbf2b5cc6d", "jobId" : "01edcd9a-553d-4117-9e98-e92b8c2aa23e", "resourceType" : "DB" } ], "description" : "update database : TEST", "cause" : "*Cause: Parameter value validation failed.", "action" : "*Action: Provide correct parameter values for the operation."

Well, now we have some idea on where the DB backup might have failed. Looking into the Oracle support, we have hit the note id OCI Managed Backup Failing with: DCS-10045:Validation error encountered: OMF parameters.OMF parameter settings are not as per DbStorage Location (Doc ID 2771233.1). Well, few days back, we changed the DB recovery DEST location from +RECO to +DATA because of space issues and that is the reason, the backups are failing. Well, that's it. I tried to give an overview on how we can troubleshoot and collect the details when the Database backups are failed. Till then, happy learning Cloud.