FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > ArchLinux > ArchLinux Pacman Development

 
 
LinkBack Thread Tools
 
Old 04-12-2012, 02:54 PM
Dave Reisner
 
Default scripts/library: introduce parseopts

This will replace our current options parser used in pacman-key and
makepkg. It follows heuristics closer to that of GNU getopt long (and
thus pacman itself), with the exception that it does not allow for
options with optional arguments. Due to the way this parser will be
used, this sort of functionality will not be needed.

Instead of relying on eval+set, options are normalized into an array,
OPTRET, which callers should expect to be populated after returning from
parseopts. This avoids problems with quotes and spaces in arguments,
assuming that the user quotes properly when passing into the
application.

A new test harness for parseopts is added in test/scripts.

Signed-off-by: Dave Reisner <dreisner@archlinux.org>
---
Makefile.am | 6 +-
configure.ac | 1 +
scripts/Makefile.am | 1 +
scripts/library/README | 12 ++++
scripts/library/parseopts.sh | 113 ++++++++++++++++++++++++++++++++++++
scripts/po/POTFILES.in | 1 +
test/scripts/Makefile.am | 9 +++
test/scripts/parseopts_test.sh | 123 ++++++++++++++++++++++++++++++++++++++++
8 files changed, 264 insertions(+), 2 deletions(-)
create mode 100644 scripts/library/parseopts.sh
create mode 100644 test/scripts/Makefile.am
create mode 100755 test/scripts/parseopts_test.sh

diff --git a/Makefile.am b/Makefile.am
index a024a2e..e08b809 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -1,4 +1,4 @@
-SUBDIRS = lib/libalpm src/util src/pacman scripts etc test/pacman test/util
+SUBDIRS = lib/libalpm src/util src/pacman scripts etc test/pacman test/util test/scripts
if WANT_DOC
SUBDIRS += doc
endif
@@ -23,7 +23,7 @@ dist_pkgdata_DATA =
proto/ChangeLog.proto

# run the pactest test suite and vercmp tests
-check-local: test/pacman test/util src/pacman src/util
+check-local: test/pacman test/scripts test/util src/pacman src/util
LC_ALL=C $(PYTHON) $(top_srcdir)/test/pacman/pactest.py --debug=1
--test $(top_srcdir)/test/pacman/tests/*.py
-p $(top_builddir)/src/pacman/pacman
@@ -31,6 +31,8 @@ check-local: test/pacman test/util src/pacman src/util
$(top_builddir)/src/util/pacsort
$(SH) $(top_srcdir)/test/util/vercmptest.sh
$(top_builddir)/src/util/vercmp
+ $(BASH_SHELL) $(top_srcdir)/test/scripts/parseopts_test.sh
+ $(top_srcdir)/scripts/library/parseopts.sh

# create the pacman DB and cache directories upon install
install-data-local:
diff --git a/configure.ac b/configure.ac
index eccf425..a6114f8 100644
--- a/configure.ac
+++ b/configure.ac
@@ -442,6 +442,7 @@ doc/Makefile
etc/Makefile
test/pacman/Makefile
test/pacman/tests/Makefile
+test/scripts/Makefile
test/util/Makefile
contrib/Makefile
Makefile
diff --git a/scripts/Makefile.am b/scripts/Makefile.am
index 328fbff..7662fba 100644
--- a/scripts/Makefile.am
+++ b/scripts/Makefile.am
@@ -27,6 +27,7 @@ EXTRA_DIST =

LIBRARY =
library/output_format.sh
+ library/parseopts.sh
library/parse_options.sh

# Files that should be removed, but which Automake does not know.
diff --git a/scripts/library/README b/scripts/library/README
index 1e9c962..33f6321 100644
--- a/scripts/library/README
+++ b/scripts/library/README
@@ -13,3 +13,15 @@ A getopt replacement to avoids portability issues, in particular the
lack of long option name support in the default getopt provided by some
platforms.
Usage: parse_option $SHORT_OPTS $LONG_OPTS "$@"
+
+parseopts.sh:
+A getopt_long-like parser which portably supports longopts and shortopts
+with some GNU extensions. It does not allow for options with optional
+arguments. For both short and long opts, options requiring an argument
+should be suffixed with a colon. Long opts must be passed comma
+separated.
+Usage:
+ parseopts "fb:z" "foo,bar:,baz" "$@"; set -- "${OPTRET[@]}"
+Returns:
+ 0: parse success
+ 1: parse failure (error message supplied)
diff --git a/scripts/library/parseopts.sh b/scripts/library/parseopts.sh
new file mode 100644
index 0000000..912bb49
--- /dev/null
+++ b/scripts/library/parseopts.sh
@@ -0,0 +1,113 @@
+# getopt-like parser
+parseopts() {
+ local opt= optarg= i= shortopts=$1
+ local -a longopts unused_argv
+
+ # split the longopt array on commas for easier parsing
+ IFS=, read -ra longopts <<< "$2"
+ shift 2
+
+ get_argreq() {
+ local o re="^($1)(:?:?)$"
+ for o in "${longopts[@]}"; do
+ # found option, return number of colons (0 = none, 1 = required)
+ [[ $o =~ $re ]] && return ${#BASH_REMATCH[2]}
+ done
+ # failure
+ return 255
+ }
+
+ while (( $# )); do
+ case $1 in
+ --) # explicit end of options
+ shift
+ break
+ ;;
+ -[!-]*) # short option
+ for (( i = 1; i < ${#1}; i++ )); do
+ opt=${1:i:1}
+
+ # option doesn't exist
+ if [[ $shortopts != *$opt* ]]; then
+ echo "@SCRIPTNAME@: $(gettext "unrecognized option") '-$opt'" >&2
+ OPTRET=(--)
+ return 1
+ fi
+
+ OPTRET+=("-$opt")
+ # option requires optarg
+ if [[ $shortopts = *$opt:* ]]; then
+ # if we're not at the end of the option chunk, the rest is the optarg
+ if (( i < ${#1} - 1 )); then
+ OPTRET+=("${1:i+1}")
+ break
+ # if we're at the end, grab the the next positional, if it exists
+ elif (( i == ${#1} - 1 )) && [[ $2 ]]; then
+ OPTRET+=("$2")
+ shift
+ break
+ # parse failure
+ else
+ printf "@SCRIPTNAME@: %s '-%s'
" "$(gettext "option %s requires an argument")" "$opt" >&2
+ OPTRET=(--)
+ return 1
+ fi
+ fi
+ done
+ ;;
+ --?*=*|--?*) # long option
+ IFS='=' read -r opt optarg <<< "${1#--}"
+ get_argreq "$opt"
+ case $? in
+ *)
+ opt=${opt%%:}
+ ;;&
+ # intentional fallthrough
+ 0)
+ if [[ $optarg ]]; then
+ printf "@SCRIPTNAME@: %s '--%s'
" "$(gettext "option %s does not allow an argument")" "$opt" >&2
+ OPTRET=(--)
+ return 1
+ else
+ OPTRET+=("--$opt")
+ shift
+ continue 2
+ fi
+ ;;
+ 1)
+ # --longopt=optarg
+ if [[ $optarg ]]; then
+ OPTRET+=("--$opt" "$optarg")
+ shift
+ # --longopt optarg
+ elif [[ $2 ]]; then
+ OPTRET+=("--$opt" "$2" )
+ shift 2
+ else
+ printf "@SCRIPTNAME@: %s '--%s'
" "$(gettext "option %s requires an argument")" "$opt" >&2
+ OPTRET=(--)
+ return 1
+ fi
+ continue 2
+ ;;
+ 255)
+ printf "@SCRIPTNAME@: %s '--%s'
" "$(gettext "unrecognized option")" "$opt" >&2
+ OPTRET=(--)
+ return 1
+ ;;
+ *)
+ printf "@SCRIPTNAME@: $(gettext "internal error: invalid option in longopt array") '--$opt'" >&2
+ esac
+ ;;
+ *) # non-option arg encountered, add it as a parameter
+ unused_argv+=("$1")
+ ;;
+ esac
+ shift
+ done
+
+ # add end-of-opt terminator and any leftover positional parameters
+ OPTRET+=('--' "${unused_argv[@]}" "$@")
+
+ return 0
+}
diff --git a/scripts/po/POTFILES.in b/scripts/po/POTFILES.in
index 007e535..01cc235 100644
--- a/scripts/po/POTFILES.in
+++ b/scripts/po/POTFILES.in
@@ -9,3 +9,4 @@ scripts/pkgdelta.sh.in
scripts/repo-add.sh.in
scripts/library/output_format.sh
scripts/library/parse_options.sh
+scripts/library/parseopts.sh
diff --git a/test/scripts/Makefile.am b/test/scripts/Makefile.am
new file mode 100644
index 0000000..b949e88
--- /dev/null
+++ b/test/scripts/Makefile.am
@@ -0,0 +1,9 @@
+check_SCRIPTS =
+ parseopts_test.sh
+
+noinst_SCRIPTS = $(check_SCRIPTS)
+
+EXTRA_DIST =
+ $(check_SCRIPTS)
+
+# vim:set ts=2 sw=2 noet:
diff --git a/test/scripts/parseopts_test.sh b/test/scripts/parseopts_test.sh
new file mode 100755
index 0000000..58f2a8e
--- /dev/null
+++ b/test/scripts/parseopts_test.sh
@@ -0,0 +1,123 @@
+#!/bin/bash
+
+declare -i testcount=0 pass=0 fail=0
+
+# source the library function
+if [[ -z $1 || ! -f $1 ]]; then
+ printf "error: path to parse_option library not provided or does not exist
"
+ exit 1
+fi
+. "$1"
+
+if ! type -t parseopts >/dev/null; then
+ printf 'parse_options function not found
'
+ exit 1
+fi
+
+# borrow opts from makepkg
+OPT_SHORT="AcdefFghiLmop:rRsV"
+OPT_LONG="allsource,asroot,ignorearch,check,clean ,nodeps"
+OPT_LONG+=",noextract,force,forcever:,geninteg,he lp,holdver"
+OPT_LONG+=",install,key:,log,nocolor,nobuild,noch eck,nosign,pkg:,rmdeps"
+OPT_LONG+=",repackage,skipinteg,sign,source,syncd eps,version,config:"
+OPT_LONG+=",noconfirm,noprogressbar"
+
+parse() {
+ local result=$1 tokencount=$2; shift 2
+
+ (( ++testcount ))
+ parseopts "$OPT_SHORT" "$OPT_LONG" "$@" 2>/dev/null
+ test_result "$result" "$tokencount" "$*" "${OPTRET[@]}"
+ unset OPTRET
+}
+
+test_result() {
+ local result=$1 tokencount=$2 input=$3; shift 3
+
+ if { [[ $result = "$*" ]] || [[ $2 = NULL && -z $1 ]]; } && (( tokencount == $# )); then
+ (( ++pass ))
+ else
+ printf '[TEST %3s]: FAIL
' "$testcount"
+ printf ' input: %s
' "$input"
+ printf ' output: %s (%s tokens)
' "$*" "$#"
+ printf ' expected: %s (%s tokens)
' "$result" "$tokencount"
+ echo
+ (( ++fail ))
+ fi
+}
+
+summarize() {
+ if (( !fail )); then
+ printf 'All %s tests successful
' "$testcount"
+ exit 0
+ else
+ printf '%s of %s tests failed
' "$fail" "$testcount"
+ exit 1
+ fi
+}
+trap 'summarize' EXIT
+
+printf 'Beginning parse_options tests
'
+
+# usage: parse <expected result> <token count> test-params...
+# a failed parse will match only the end of options marker '--'
+
+# no options
+parse '--' 1
+
+# short options
+parse '-s -r --' 3 -s -r
+
+# short options, no spaces
+parse '-s -r --' 3 -sr
+
+# short opt missing an opt arg
+parse '--' 1 -s -p
+
+# short opt with an opt arg
+parse '-p PKGBUILD -L --' 4 -p PKGBUILD -L
+
+# short opt with an opt arg, no space
+parse '-p PKGBUILD --' 3 -pPKGBUILD
+
+# valid shortopts as a long opt
+parse '--' 1 --sir
+
+# long opt wiht no optarg
+parse '--log --' 2 --log
+
+# long opt with missing optarg
+parse '--' 1 -sr --pkg
+
+# long opt with optarg
+parse '--pkg foo --' 3 --pkg foo
+
+# long opt with optarg with whitespace
+parse '--pkg foo bar -- baz' 4 --pkg "foo bar" baz
+
+# long opt with optarg with =
+parse '--pkg foo=bar -- baz' 4 --pkg foo=bar baz
+
+# long opt with explicit optarg
+parse '--pkg bar -- foo baz' 5 foo --pkg=bar baz
+
+# long opt with explicit optarg, with whitespace
+parse '--pkg foo bar -- baz' 4 baz --pkg="foo bar"
+
+# long opt with explicit optarg that doesn't take optarg
+parse '--' 1 --force=always -s
+
+# long opt with explicit optarg with =
+parse '--pkg foo=bar --' 3 --pkg=foo=bar
+
+# explicit end of options with options after
+parse '-s -r -- foo bar baz' 6 -s -r -- foo bar baz
+
+# non-option parameters mixed in with options
+parse '-s -r -- foo baz' 5 -s foo baz -r
+
+# optarg with whitespace
+parse '-p foo bar -s --' 4 -p'foo bar' -s
+
+# non-option parameter with whitespace
+parse '-i -- foo bar' 3 -i 'foo bar'
--
1.7.10
 
Old 04-13-2012, 04:46 AM
Allan McRae
 
Default scripts/library: introduce parseopts

On 13/04/12 00:54, Dave Reisner wrote:
> This will replace our current options parser used in pacman-key and
> makepkg. It follows heuristics closer to that of GNU getopt long (and
> thus pacman itself), with the exception that it does not allow for
> options with optional arguments. Due to the way this parser will be
> used, this sort of functionality will not be needed.
>
> Instead of relying on eval+set, options are normalized into an array,
> OPTRET, which callers should expect to be populated after returning from
> parseopts. This avoids problems with quotes and spaces in arguments,
> assuming that the user quotes properly when passing into the
> application.
>
> A new test harness for parseopts is added in test/scripts.
>
> Signed-off-by: Dave Reisner <dreisner@archlinux.org>
> ---

<snip>

> +
> + get_argreq() {
> + local o re="^($1)(:?:?)$"

You only want one :? in the regex.

> + for o in "${longopts[@]}"; do
> + # found option, return number of colons (0 = none, 1 = required)
> + [[ $o =~ $re ]] && return ${#BASH_REMATCH[2]}
> + done
> + # failure
> + return 255
> + }

<snip>

> +test_result() {
> + local result=$1 tokencount=$2 input=$3; shift 3
> +
> + if { [[ $result = "$*" ]] || [[ $2 = NULL && -z $1 ]]; } && (( tokencount == $# )); then

What is the "|| [[ $2 = NULL && -z $1 ]];" part for? I can guess, but
it seems unneeded.


With the minor changes I pointed out in the last few emails, I ack this
patchset. My biggest complaint is I prefer the name "parse_options"
over "parseopts"!

Allan
 
Old 04-13-2012, 12:35 PM
Dave Reisner
 
Default scripts/library: introduce parseopts

On Fri, Apr 13, 2012 at 12:46 AM, Allan McRae <allan@archlinux.org> wrote:

> On 13/04/12 00:54, Dave Reisner wrote:
> > This will replace our current options parser used in pacman-key and
> > makepkg. It follows heuristics closer to that of GNU getopt long (and
> > thus pacman itself), with the exception that it does not allow for
> > options with optional arguments. Due to the way this parser will be
> > used, this sort of functionality will not be needed.
> >
> > Instead of relying on eval+set, options are normalized into an array,
> > OPTRET, which callers should expect to be populated after returning from
> > parseopts. This avoids problems with quotes and spaces in arguments,
> > assuming that the user quotes properly when passing into the
> > application.
> >
> > A new test harness for parseopts is added in test/scripts.
> >
> > Signed-off-by: Dave Reisner <dreisner@archlinux.org>
> > ---
>
> <snip>
>
> > +
> > + get_argreq() {
> > + local o re="^($1)(:?:?)$"
>
> You only want one :? in the regex.
>

Ah yes.. a vestigial from when I was toying with optional arguments.


>
> > + for o in "${longopts[@]}"; do
> > + # found option, return number of colons (0 = none,
> 1 = required)
> > + [[ $o =~ $re ]] && return ${#BASH_REMATCH[2]}
> > + done
> > + # failure
> > + return 255
> > + }
>
> <snip>
>
> > +test_result() {
> > + local result=$1 tokencount=$2 input=$3; shift 3
> > +
> > + if { [[ $result = "$*" ]] || [[ $2 = NULL && -z $1 ]]; } && ((
> tokencount == $# )); then
>
> What is the "|| [[ $2 = NULL && -z $1 ]];" part for? I can guess, but
> it seems unneeded.
>
>
I have no idea what it's for. This test toy (and some of the parseopts
code) dates back to July-ish of last year. Removing the entire middle check
seems innocuous, so its gone.


>
> With the minor changes I pointed out in the last few emails, I ack this
> patchset. My biggest complaint is I prefer the name "parse_options"
> over "parseopts"!
>
> Allan
>
 

Thread Tools




All times are GMT. The time now is 11:26 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org