Some how I got my CloudForms SystemEngine into a weird state. I unregistered a consumer while simultaneously removing it from the Katello UI. This caused a race condition because the UI got a 410 GONE error from Candlepin and it didn’t remove it from its database. Clearly a bug but still that left me in a state where the Systems page would show the 410 ERROR and leave me with a spinning cursor.

As this was simply a test setup I could’ve just reinstalled. But I wanted to know how this works. As you know Katello is made of many parts: Pulp, Candlepin, and Foreman.

My particular installation didn’t have Foreman, so that left: Pulp, Candlepin, Katello proper, and Elasticsearch. Since we were getting a 410 from Candlepin that means the consumer was already gone from that database

cp_consumer

table.

First, I removed the system from Katello’s database.

# psql -U katellouser katelloschema
katelloschema=> delete from systems where id = 1;
katelloschema=> commit;
katelloschema=> \q

Then I removed the consumer from Pulp:

# mongo
MongoDB shell version: 1.8.2
connecting to: test
> show dbs
admin	(empty)
local	(empty)
pulp_database	0.203125GB
> use pulp_database
switched to db pulp_database
> show collections
...
consumers
...
> db.consumers.remove()
> 

What in the world did you just do?🙂 We can see that pulp uses the

pulp_database

. And that one of the collections was

consumers

. Running

db.consumers.help()

revealed there was a remove method. In my case I only had one system so I just did

db.consumers.remove()

. But if you have more you’ll want to specify the key:

db.consumers.remove({"_id":"62d61426-3cf9-4afb-bc3b-e0f696950adc"})

. Ok we’re done. Let’s go checkout Katello again.

UGH Still shows spinning cursor. So what else could be pointing to the fact that we had a system? Elasticsearch which is the search engine for Katello. That just needs reindexing.

# cd /usr/share/katello
# rake reindex
Search Indices cleared.
Re-indexing ActivationKey
Re-indexing Changeset
Re-indexing Filter
Re-indexing GpgKey
Re-indexing Notice
Re-indexing Organization
Re-indexing Provider
Re-indexing Role
Re-indexing SyncPlan
Re-indexing System
Re-indexing TaskStatus
Re-indexing User
Re-indexing Repositories
# 

Ok let’s try one more time. Visit your systems page in Katello and now you should see that there is no longer a spinning cursor, but a blank list of systems. This is what I expected. YAY!

4 thoughts on “Cleaning up failed system delete in Katello

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s