Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
W
Webpage
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Package Registry
Container Registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
LEAPS WG3
Webpage
Commits
5c7f3cb6
Commit
5c7f3cb6
authored
3 months ago
by
Paul Millar
Browse files
Options
Downloads
Patches
Plain Diff
update_oai-pmh use standard function to emit logging line
parent
21a17caf
No related branches found
Branches containing commit
No related tags found
1 merge request
!64
update_oai-pmh Add a back-off strategy, to allow a service to recover
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
scripts/update_oai-pmh.rb
+13
-8
13 additions, 8 deletions
scripts/update_oai-pmh.rb
with
13 additions
and
8 deletions
scripts/update_oai-pmh.rb
+
13
−
8
View file @
5c7f3cb6
...
...
@@ -16,26 +16,26 @@ def check_oai_pmh_endpoint(endpoint_url)
response
=
HTTParty
.
head
(
queryIdentify_url
)
if
!
response
.
success?
p
uts
"
Error: Identify response has HTTP status code
#{
response
.
code
}
."
p
rint_with_time
(
"
Error: Identify response has HTTP status code
#{
response
.
code
}
."
)
return
"Error"
end
response
=
HTTParty
.
get
(
queryIdentify_url
)
if
response
.
body
.
nil?
||
response
.
body
.
empty?
p
uts
"
Error: Identify response is empty."
p
rint_with_time
(
"
Error: Identify response is empty."
)
return
"Error"
end
xml_response
=
Nokogiri
::
XML
(
response
.
body
)
oai_pmh_tag
=
xml_response
.
at_xpath
(
'//*[name()="OAI-PMH"]'
)
if
!
oai_pmh_tag
p
uts
"
Error: Identify response has no OAI-PMH tag."
p
rint_with_time
(
"
Error: Identify response has no OAI-PMH tag."
)
return
"Error"
end
return
"Active"
rescue
StandardError
=>
e
p
uts
"
Error: Identify request failed:
#{
e
.
message
}
"
p
rint_with_time
(
"
Error: Identify request failed:
#{
e
.
message
}
"
)
return
"Error"
end
end
...
...
@@ -72,7 +72,7 @@ def list_sets(endpoint)
end
end
rescue
StandardError
=>
e
p
uts
"
Error: ListSets request failed:
#{
e
.
message
}
"
p
rint_with_time
(
"
Error: ListSets request failed:
#{
e
.
message
}
"
)
end
return
sets_info
end
...
...
@@ -131,7 +131,7 @@ def list_identifiers(url)
if
hasServerResponded
&&
attempts
==
1
print
"
\n
"
end
print
"
Attempt
#{
attempts
}
of
#{
MAX_HTTP_GET_RETRIES
}
failed:
#{
e
.
message
}
\n
"
print
_with_time
(
"
Attempt
#{
attempts
}
of
#{
MAX_HTTP_GET_RETRIES
}
failed:
#{
e
.
message
}
"
)
if
attempts
>=
MAX_HTTP_GET_RETRIES
raise
StandardError
.
new
"Too many GET requests failed for
#{
url
}
"
...
...
@@ -191,6 +191,10 @@ def list_identifiers(url)
return
total
,
set_counts
,
resumptionToken
end
def
print_with_time
(
text
)
print
"
#{
Time
.
now
.
strftime
(
"%d/%m/%Y %H:%M:%S"
)
}
#{
text
}
\n
"
end
def
count_identifiers
(
endpoint
,
prefix
)
url
=
endpoint
+
"?verb=ListIdentifiers&metadataPrefix="
+
prefix
...
...
@@ -200,7 +204,8 @@ def count_identifiers(endpoint, prefix)
while
resumptionToken
if
i
%
40
==
0
if
i
>
0
print
"
\n
#{
Time
.
now
.
strftime
(
"%d/%m/%Y %H:%M:%S"
)
}
harvested
#{
overall_total
}
records so far.
\n
"
print
(
"
\n
"
)
print_with_time
(
"harvested
#{
overall_total
}
records so far."
)
end
print
" Harvesting: "
end
...
...
@@ -236,7 +241,7 @@ def query_oai_pmh_endpoint(endpoint)
begin
total_count
,
set_counts
=
count_identifiers
(
endpoint
,
dc_prefix
)
rescue
StandardError
=>
e
p
uts
"
Error: ListIdentifiers failed:
#{
e
.
message
}
"
p
rint_with_time
(
"
Error: ListIdentifiers failed:
#{
e
.
message
}
"
)
total_count
=
0
set_counts
=
{}
end
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment