Unless there's a clear performance or portability advantage, generated code is just another place where things can go wrong, and one that's harder to track down at that.
Well, if you are concerned about well-debugged tools breaking then we might as well all go back to writing machine code directly - after all compilers produce 'generated code'
As I've said before, the .lcidl is not a blocker - just an unneeded extra hurdle where Java reflection suffices
This is making the assumption that Java reflection *does* suffice for mapping LiveCode level types to 'native' code (whether that be C++, Obj-C, Java or any other language). Java does provide a significant amount of information through its reflection process, but there are other things that the (replacement for) lcidl spec files will allow you to specify that goes outside the scope of any reflection process because they are related to LiveCode syntax and typing. It might be that an annotation processor could be written - but then isn't that another tool that could go wrong when we will (eventually) have one that performs the function for all language bindings?
I think what you want in the unified externals api is already there in the interface and we could do what we like with it. We aren't required to use the lcidl compiler and all the details of the interface are there in a generated file and the engine.
The only supported mechanism for accessing the externals functionality at the moment is through the v0 interface and through the lcidl mechanism - the v1 interface is too inherently tied to the engine internals and will be gone in the refactored engine (so if you want what you write now to work moving forward, then don't touch the raw v1 interface!)... The access to variables it exposes is too low-level and means it can't be changed to use the new reference counted typing system that is being introduced without a shim that completely copies arrays and the like - this is something we are trying to move away from to ensure that there is no performance penalty anywhere we can avoid it. (By the way, in terms of forward-compatibility for existing iOS externals, lcidlc will just be changed so that it generates the new specification information rather than generating code; and for forward-compatibility for v0 externals, a shim will be written that emulates the v0 interface although that will be immediately deprecated).
An important point to heed is that 'externals' in their current form are going away - *all* engine functionality is being refactored into distinct modules where the functionality is completely separated from all LiveCode-script level concerns. The bindings of these modules to script will be through a specification file. When this work is completed externals as they are now will be no different from any other module - they won't be second-class citizens.
Essentially a module will consist of a compiled specification (note this will be a serialized data structure rather than any generated code) and the implementation along with module-specific resources. The implementation will not have to be all in one language, but will be able to be a melange - for example a external exposing mobile functionality might be written in Obj-C for iOS, Java for Android and LiveCode for the other platforms (the latter perhaps just providing no-ops, or a simple emulation). The engine will use the information in the specification to work out what the syntax is for the module, what method (in whatever language - I'm using 'method' here as an umbrella term to describe a function/handler/static method/etc. that is present in the implementation at the level below LiveCode script) to call when and with what (native) types. The goal is that the only code you have to write is code directly related to providing the functionality - in particular, no messing about with type checking or type conversion.
Originally I had planned for 'just' using the typing information from the implementation language to define (at least) the bindings to syntax. However, this is woefully insufficient - after a fair bit of analysis and pondering it became clear that the LiveCode 'type environment' is a great deal richer than just strings, numbers and arrays. Indeed, here is a summary of the current range of types that LiveCode syntax we currently have actually uses:
- The type ecosystem at the script level naturally splits up into the following collection of types:
- any - the umbrella type, which can be anything
- boolean - a type consisting of two values true or false
- number - a real or integral number
- integer - an integral number
- unsigned integer - an integral number >= 0
- character - a string of length ‘1’ (essentially a Unicode codepoint or native equivalent)
- byte - a binary string of length 1
- string - an arbitrarily long sequence of characters
- binary string - an arbitrarily long sequence of bytes
- map - an associative mapping between strings and values of any type (here the keys are considered to be case-sensitive)
- dictionary - an associative mapping between names and values of any type (here the keys are considered to be case-preserving but case-insensitive)
- array - a sequence of values
- items - a sequence of strings separated by ‘,’
- lines - a sequence of strings separated by the return character
- enum - a family of types where for each type a value can be one of a fixed number of strings
- set - a family of types where for each type a value can be a collection of a fixed number of strings
- record - a family of types where for each type a value is a mapping from a fixed set of strings to values of any type
- variant - a family of types where for each type a value can be one a of a fixed set of other types
Most of these types can also be annotated to make them more specific. In particular:
- numeric types can be restricted to a particular range
- sequence and array types can be restricted to hold values of only a specific type
- sequence types can be restricted to be of a certain length
- record types can be restricted so that specific elements can only be of a specific type
In addition to this there is also the idea of an 'opaque' type, which represents something which has no script-level representation (and thus is only used as the result of methods and immediately a parameter of another method), or something which has a mapping to some other script type but is much better presented to the 'native' method as a 'native' type. (There's also another type waiting to be added to the list the 'tuple' - a comma-separated list of values of differing types - we noticed this yesterday whilst going reviewing some things).
Now, as it stands for the version of the engine that will appear before we replace the parser (we're aiming to get an engine which has 'identical' functionality to now, but using the new split between evaluation of parameters and implementation methods) the specification files will not contain syntax (yet), just a description of the methods a module exposes from the point of view of LiveCode script types. For example, here is the spec for the functionality provided by what-will-be the 'printing' module (note this is for exposition, it isn't quite yet final - also this module doesn't use out/inout parameters nor some of the more complicated types, such as records):
Code: Select all
module Printing
import Interface.Point
import Interface.Rectangle
import Interface.Stack
import Interface.Card
set PrinterFeatures
"collate" is Collate
"copies" is Copies
"color" is Color
"duplex" is Duplex
end set
enum PrinterPageOrientation
"portrait" is Portrait
"reverse portrait" is ReversePortrait
"landscape" is Landscape
"reverse landscape" is ReverseLandscape
end enum
enum PrinterJobDuplex
"none" is Simplex
"short edge" is ShortEdge
"long edge" is LongEdge
end enum
enum PrinterLinkType
"" is Unspecified
"anchor" is Anchor
"url" is URI
end enum
enum PrinterBookmarkInitialStateType
"open" is true
"closed" is false
end enum
opaque PrinterPageRange
"" is All
"all" is All
"current" is Current
"selection" is Selection
string is Custom
end opaque
opaque PrinterDeviceOutput
"device" is Device
"preview" is Preview
"system" is System
string is Custom
end opaque
command AnswerPageSetup(in as_sheet is boolean)
command AnswerPrinter(in as_sheet is boolean)
command OpenPrinting()
command OpenPrintingWithDialog(in as_sheet is boolean)
command OpenPrintingToDestination(in destination is string, in filename is string, in options is optional array)
command ClosePrinting()
command CancelPrinting()
command ResetPrinting()
command PrintAnchor(in name is string, in location is Point)
command PrintLink(in type is PrinterLinkType, in area is rectangle)
command PrintNativeBookmark(in title is string, in location is Point, in level is unsigned integer, in initially_closed is boolean)
command PrintUnicodeBookmark(in title is binary string, in location is Point, in level is unsigned integer, in initially_closed is boolean)
command PrintBreak()
command PrintAllCards(in target is optional Stack, in only_marked is boolean)
command PrintRectOfAllCards(in target is optional Stack, in only_marked is boolean, in from is Point, in to is Point)
command PrintCard(in target is optional Card)
command PrintRectOfCard(in target is optional Card, in from is Point, in to is Point)
command PrintSomeCards(in count is unsigned integer)
command PrintRectOfSomeCards(in count is unsigned integer, in from is Point, in to is Point)
command PrintRectOfCardIntoRect(in target is optional Card, in src_from is Point, in src_to is Point, in dst_rect is Rectangle)
global readonly property PrintDeviceFeatures is PrinterFeatures
global property PrintDeviceOutput is PrinterDeviceOutput
global property PrintPageOrientation is PrinterPageOrientation
global property PrintJobRanges is PrinterPageRange
global property PrintPageSize is items(2) of integer
global property PrintPageScale is number
global readonly property PrintPageRectangle is Rectangle
global property PrintJobName is string
global property PrintJobCopies is integer
global property PrintJobCollate is boolean
global property PrintJobColor is boolean
global readonly property PrintJobPage is integer
global property PrintJobDuplex is PrinterJobDuplex
global readonly property PrintDeviceRectangle is Rectangle
global property PrintDeviceSettings is binary string
global property PrintDeviceName is string
global property PrintCardBorders is boolean
global property PrintGutters is items(2) of integer
global property PrintMargins is items(4) of integer
global property PrintRowsFirst is boolean
global property PrintScale is number
global property PrintRotated is boolean
global property PrintCommand is string
global property PrintFontTable is string
global readonly property PrinterNames is string
end module
So, the specification file above provides a language-neutral (from an implementation point of view) description of how script will interact with the implementation methods. The other piece of the puzzle is the type information from the implementation - now, in Java this will be extractable using the reflection mechanism, however as C/C++ doesn't provide this ability we've written a tool (that uses gccxml) that parses a header file for the module and generates the necessary information (note this is for exposition, it isn't quite yet final):
Code: Select all
#ifndef __MC_EXEC_PRINTING__
#define __MC_EXEC_PRINTING__
#ifndef __MC_SYSDEFS__
#include “sysdefs.h”
#endif
#ifndef __MC_INTERFACE__
#include “interface.h”
#endif
typedef intset_t MCPrintingPrinterFeatures;
enum /* MCPrintingPrinterFeatures */
{
kMCPrintingPrinterFeatureCollate,
kMCPrintingPrinterFeatureCopies,
kMCPrintingPrinterFeatureColor,
kMCPrintingPrinterFeatureDuplex,
};
enum MCPrintingPrinterPageOrientation
{
kMCPrintingPrinterPageOrientationPortrait,
kMCPrintingPrinterPageOrientationReversePortrait,
kMCPrintingPrinterPageOrientationLandscape,
kMCPrintingPrinterPageOrientationReverseLandscape,
};
enum MCPrintingPrinterJobDuplex
{
kMCPrintingPrinterJobDuplexNone,
kMCPrintingPrinterJobDuplexShortEdge,
kMCPrintingPrinterJobDuplexLongEdge,
};
enum MCPrintingPrinterLinkType
{
kMCPrintingPrinterLinkTypeUnspecified,
kMCPrintingPrinterLinkTypeAnchor,
kMCPrintingPrinterLinkTypeURI,
};
//////////
enum MCPrintingPrinterPageRangeRepType
{
kMCPrintingPrinterPageRangeRepTypeAll,
kMCPrintingPrinterPageRangeRepTypeCurrent,
kMCPrintingPrinterPageRangeRepTypeSelection,
kMCPrintingPrinterPageRangeRepTypeCustom,
};
struct MCPrintingPrinterPageRangeRep
{
MCPrintingPrinterPageRangeRepType type;
union
{
MCStringRef custom;
};
};
struct MCPrintingPrinterPageRange;
void MCPrintingPrinterPageRangeFree(MCExecContext& ctxt, MCPrintingPrinterPageRange& value);
void MCPrintingPrinterPageRangeCopy(MCExecContext& ctxt, MCPrintingPrinterPageRange& src_value, MCPrintingPrinterPageRange& dst_value);
void MCPrintingPrinterPageRangeEncode(MCExecContext& ctxt, MCPrintingPrinterPageRangeRep& rep, MCPrintingPrinterPageRange& value, MCExecErrorInfo*& r_error);
void MCPrintingPrinterPageRangeDecode(MCExecContext& ctxt, MCPrintingPrinterPageRange& value, MCPrintingPrinterPageRangeRep& rep);
//////////
enum MCPrintingPrinterDeviceOutputRepType
{
kMCPrintingPrinterDeviceOutputRepTypeDevice,
kMCPrintingPrinterDeviceOutputRepTypePreview,
kMCPrintingPrinterDeviceOutputRepTypeSystem,
kMCPrintingPrinterDeviceOutputRepTypeCustom,
};
struct MCPrintingPrinterDeviceOutputRep
{
MCPrintingPrinterDeviceOutputRepType type;
union
{
MCStringRef custom;
};
};
struct MCPrintingPrinterDeviceOutput;
void MCPrintingPrinterDeviceOutputFree(MCExecContext& ctxt, MCPrintingPrinterDeviceOutput& value);
void MCPrintingPrinterPageRangeCopy(MCExecContext& ctxt, const MCPrintingPrinterDeviceOutput& src_value, MCPrintingPrinterDeviceOutput& dst_value);
void MCPrintingPrinterPageRangeEncode(MCExecContext& ctxt, MCPrintingPrinterDeviceOutputRep& rep, MCPrintingPrinterDeviceOutput& value, MCExecErrorInfo*& r_error);
void MCPrintingPrinterPageRangeDecode(MCExecContext& ctxt, const MCPrintingPrinterDeviceOutput& value, MCPrintingPrinterDeviceOutputRep& rep);
//////////
void MCPrintingExecAnswerPageSetup(MCExecContext &ctxt, bool p_is_sheet);
void MCPrintingExecAnswerPrinter(MCExecContext &ctxt, bool p_is_sheet);
void MCPrintingExecCancelPrinting(MCExecContext& ctxt);
void MCPrintingExecResetPrinting(MCExecContext& ctxt);
void MCPrintingExecPrintAnchor(MCExecContext& ctxt, MCStringRef name, MCPoint location);
void MCPrintingExecPrintLink(MCExecContext& ctxt, int type, MCStringRef target, MCRectangle area);
void MCPrintingExecPrintNativeBookmark(MCExecContext& ctxt, MCStringRef title, MCPoint location, integer_t level, bool initially_closed);
void MCPrintingExecPrintUnicodeBookmark(MCExecContext& ctxt, MCStringRef title, MCPoint location, integer_t level, bool initially_closed);
void MCPrintingExecPrintBreak(MCExecContext& ctxt);
void MCPrintingExecPrintAllCards(MCExecContext& ctxt, MCStack *stack, bool only_marked);
void MCPrintingExecPrintRectOfAllCards(MCExecContext& ctxt, MCStack *stack, bool p_only_marked, MCPoint from, MCPoint to);
void MCPrintingExecPrintCard(MCExecContext& ctxt, MCCard *target);
void MCPrintingExecPrintRectOfCard(MCExecContext& ctxt, MCCard *target, MCPoint from, MCPoint to);
void MCPrintingExecPrintSomeCards(MCExecContext& ctxt, integer_t count);
void MCPrintingExecPrintRectOfSomeCards(MCExecContext& ctxt, integer_t count, MCPoint from, MCPoint to);
void MCPrintingExecPrintCardIntoRect(MCExecContext& ctxt, MCCard *card, MCRectangle destination);
void MCPrintingExecPrintRectOfCardIntoRect(MCExecContext& ctxt, MCCard *card, MCPoint from, MCPoint to, MCRectangle destination);
void MCPrintingExecClosePrinting(MCExecContext& ctxt);
void MCPrintingExecOpenPrintingToDestination(MCExecContext& ctxt, MCStringRef p_destination, MCStringRef p_filename, MCArrayRef p_options);
void MCPrintingExecOpenPrinting(MCExecContext& ctxt);
void MCPrintingExecOpenPrintingWithDialog(MCExecContext& ctxt, bool p_as_sheet);
void MCPrintingGetPrinterNames(MCExecContext& ctxt, MCStringRef& r_printers);
void MCPrintingGetPrintDeviceFeatures(MCExecContext& ctxt, MCPrintingPrinterFeatures& r_features);
void MCPrintingSetPrintDeviceOutput(MCExecContext& ctxt, const MCPrintingPrinterDeviceOutput& output);
void MCPrintingGetPrintDeviceOutput(MCExecContext& ctxt, MCPrintingPrinterDeviceOutput& r_output);
void MCPrintingGetPrintDeviceRectangle(MCExecContext& ctxt, MCRectangle &r_rectangle);
void MCPrintingGetPrintDeviceRectangle(MCExecContext& ctxt, MCRectangle &r_rectangle);
void MCPrintingGetPrintDeviceSettings(MCExecContext& ctxt, MCStringRef &r_settings);
void MCPrintingSetPrintDeviceSettings(MCExecContext& ctxt, MCStringRef p_settings);
void MCPrintingGetPrintDeviceName(MCExecContext& ctxt, MCStringRef &r_name);
void MCPrintingSetPrintDeviceName(MCExecContext& ctxt, MCStringRef p_name);
void MCPrintingGetPrintPageOrientation(MCExecContext& ctxt, MCPrintingPrinterPageOrientation& r_orientation);
void MCPrintingSetPrintPageOrientation(MCExecContext& ctxt, MCPrintingPrinterPageOrientation orientation);
void MCPrintingSetPrintJobRanges(MCExecContext& ctxt, const MCPrintingPrinterPageRange& p_ranges);
void MCPrintingGetPrintJobRanges(MCExecContext& ctxt, MCPrintingPrinterPageRange& r_ranges);
void MCPrintingSetPrintPageSize(MCExecContext& ctxt, integer_t p_value[2]);
void MCPrintingGetPrintPageSize(MCExecContext& ctxt, integer_t r_value[2]);
void MCPrintingSetPrintPageScale(MCExecContext& ctxt, double p_value);
void MCPrintingGetPrintPageScale(MCExecContext& ctxt, double &r_value);
void MCPrintingGetPrintPageRectangle(MCExecContext& ctxt, MCRectangle &r_value);
void MCPrintingGetPrintJobName(MCExecContext& ctxt, MCStringRef &r_value);
void MCPrintingSetPrintJobName(MCExecContext& ctxt, MCStringRef p_value);
void MCPrintingGetPrintJobCopies(MCExecContext& ctxt, integer_t &r_value);
void MCPrintingSetPrintJobCopies(MCExecContext& ctxt, integer_t p_value);
void MCPrintingGetPrintJobDuplex(MCExecContext& ctxt, MCPrintingPrintJobDuplex& r_value);
void MCPrintingSetPrintJobDuplex(MCExecContext& ctxt, MCPrintingPrintJobDuplex p_value);
void MCPrintingGetPrintJobCollate(MCExecContext& ctxt, bool &r_value);
void MCPrintingSetPrintJobCollate(MCExecContext& ctxt, bool p_value);
void MCPrintingGetPrintJobColor(MCExecContext& ctxt, bool &r_value);
void MCPrintingSetPrintJobColor(MCExecContext& ctxt, bool p_value);
void MCPrintingGetPrintJobPage(MCExecContext& ctxt, integer_t &r_value);
void MCPrintingGetPrintCardBorders(MCExecContext& ctxt, bool &r_card_borders);
void MCPrintingSetPrintCardBorders(MCExecContext& ctxt, bool p_card_borders);
void MCPrintingGetPrintGutters(MCExecContext& ctxt, integer_t r_gutters[2]);
void MCPrintingSetPrintGutters(MCExecContext& ctxt, integer_t p_gutters[2]);
void MCPrintingGetPrintMargins(MCExecContext& ctxt, integer_t r_margins[4]);
void MCPrintingSetPrintMargins(MCExecContext& ctxt, integer_t p_margins[4]);
void MCPrintingGetPrintRowsFirst(MCExecContext& ctxt, bool &r_rows_first);
void MCPrintingSetPrintRowsFirst(MCExecContext& ctxt, bool p_rows_first);
void MCPrintingGetPrintScale(MCExecContext& ctxt, double &r_scale);
void MCPrintingSetPrintScale(MCExecContext& ctxt, double p_scale);
void MCPrintingGetPrintRotated(MCExecContext& ctxt, bool &r_rotated);
void MCPrintingSetPrintRotated(MCExecContext& ctxt, bool p_rotated);
void MCPrintingGetPrintCommand(MCExecContext& ctxt, MCStringRef &r_command);
void MCPrintingSetPrintCommand(MCExecContext& ctxt, MCStringRef p_command);
void MCPrintingGetPrintFontTable(MCExecContext& ctxt, MCStringRef &r_table);
void MCPrintingSetPrintFontTable(MCExecContext& ctxt, MCStringRef p_table);
#endif
With this information (suitable encoded), the engine can then work out how to map between the (abstract, implementation language-neutral) script-level definition of the method to the actual implementation - in whatever language it might be.
(I perhaps should stress that the above example doesn't really show the full richness of the type mapping we are going for - for example, native methods will be able to specify parameters as being things like 'MCMutableStringRef', which can map to an 'inout string' parameter at the script-level. This means that, wherever possible, that method will get (essentially) direct access to a variable's contents to modify - there won't need to be any copying.)
Over time the specification file will be expanded to not just hold the method descriptions, but also the syntax that will be used to call them and reference documentation for the syntax itself.
Just paying reference to the desire for 'pure Java externals with no external tools needed' then, yes, eventually you probably could produce a tool that generated the necessary information from class files - although I do think that relying on annotations would make things much harder to read than a cohesive 'LiveCode-script level' spec (particularly for the structured entities that you can specify, and for syntax which will in many cases required auxillary clauses)... Also given that other aspects of Java development (like C/C++ etc.) require tools to process interface definitions (in particular if you want to interact with CORBA - you write an IDL file and use idlj) - there is already precedent. In the end, if the result of writing a specification file and having to compile it in some way gives you a great deal more than you could get without it then I don't think anyone is going to mind too much.